Original: A NTI -S YMMETRIC DGN: A STABLE ARCHITECTURE FOR D EEP G RAPH N ETWORKS
Revised: A NTI -S YMMETRIC DGN: A STABLE ARCHITECTURE FOR D EEP G RAPH N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep Graph Networks (DGNs) currently dominate the research landscape of learning from graphs, due to their efficiency and ability to implement an adaptive message-passing scheme between the nodes.
Revised: Deep Graph Networks (DGNs) currently dominate the research landscape of learning from graphs, due to their efficiency and ability to implement an adaptive message-passing scheme between the nodes.

Original: However, DGNs are typically limited in their ability to propagate and preserve long-term dependencies between nodes, i.e., they suffer from the over-squashing phenomena.
Revised: However, DGNs are typically limited in their ability to propagate and preserve long-term dependencies between nodes, i.e., they suffer from the over-squashing phenomena.

Original: This reduces their effectiveness, since predictive problems may require to capture interactions at different, and possibly large, radii in order to be effectively solved.
Revised: This reduces their effectiveness, since predictive problems may require to capture interactions at different, and possibly large, radii in order to be effectively solved.

Original: In this work, we present Anti-Symmetric Deep Graph Networks (A-DGNs), a framework for stable and non-dissipative DGN design, conceived through the lens of ordinary differential equations.
Revised: In this work, we present Anti-Symmetric Deep Graph Networks (A-DGNs), a framework for stable and non-dissipative DGN design, conceived through the lens of ordinary differential equations.

Original: We give theoretical proof that our method is stable and non-dissipative, leading to two key results: long-range information between nodes is preserved, and no gradient vanishing or explosion occurs in training.
Revised: We give theoretical proof that our method is stable and non-dissipative, leading to two key results: long-range information between nodes is preserved, and no gradient vanishing or explosion occurs in training.

Original: We empirically validate the proposed approach on several graph benchmarks, showing that A-DGN leads to improved performance and enables to learn effectively even when dozens of layers are used.
Revised: We empirically validate the proposed approach on several graph benchmarks, showing that A-DGN leads to improved performance and enables to learn effectively even when dozens of layers are used.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Representation learning for graphs has become one of the most prominent fields in machine learning.
Revised: Representation learning for graphs has become one of the most prominent fields in machine learning.

Original: Such popularity derives from the ubiquitousness of graphs.
Revised: Such popularity derives from the ubiquitousness of graphs.

Original: Indeed, graphs are an extremely powerful tool to represent systems of relations and interactions and are extensively employed in many domains (Battaglia et al., 2016;
Revised: Indeed, graphs are an extremely powerful tool to represent systems of relations and interactions and are extensively employed in many domains (Battaglia et al., 2016;

Original: Gilmer et al., 2017; Zitnik et al., 2018;
Revised: Gilmer et al., 2017; Zitnik et al., 2018;

Original: Monti et al., 2019;
Revised: Monti et al., 2019;

Original: Derrow-Pinion et al., 2021).
Revised: Derrow-Pinion et al., 2021).

Original: For example, they can model social networks, molecular structures, protein-protein interaction networks, recommender systems, and traffic networks.
Revised: For example, they can model social networks, molecular structures, protein-protein interaction networks, recommender systems, and traffic networks.

Original: The primary challenge in this field is how we capture and encode structural information in the learning model.
Revised: The primary challenge in this field is how we capture and encode structural information in the learning model.

Original: Common methods used in representation learning for graphs usually employ Deep
Revised: Common methods used in representation learning for graphs usually employ Deep

Original: Graph Networks (DGNs) (Bacciu et al., 2020; Wu et al., 2021).
Revised: Graph Networks (DGNs) (Bacciu et al., 2020; Wu et al., 2021).

Original: DGNs are a family of learning models that learn a mapping function that compresses the complex relational information encoded in a graph into an information-rich feature vector that reflects both the topological and the label information in the original graph.
Revised: DGNs are a family of learning models that learn a mapping function that compresses the complex relational information encoded in a graph into an information-rich feature vector that reflects both the topological and the label information in the original graph.

Original: As widely popular with neural networks, also DGNs consists of multiple layers.
Revised: As widely popular with neural networks, also DGNs consists of multiple layers.

Original: Each of them updates the node representations by aggregating previous node states and their neighbors, following a message passing paradigm.
Revised: Each of them updates the node representations by aggregating previous node states and their neighbors, following a message passing paradigm.

Original: However, in some problems, the exploitation of local interactions between nodes is not enough to learn representative embeddings.
Revised: However, in some problems, the exploitation of local interactions between nodes is not enough to learn representative embeddings.

Original: In this scenario, it is often the case that the DGN needs to capture information concerning interactions between nodes that are far away in the graph, i.e., by stacking multiple layers.
Revised: In this scenario, it is often the case that the DGN needs to capture information concerning interactions between nodes that are far away in the graph, i.e., by stacking multiple layers.

Original: A specific predictive problem typically needs to consider a specific range of node interactions in order to be effectively solved, hence requiring a specific number (possibly large) of DGN layers.
Revised: A specific predictive problem typically needs to consider a specific range of node interactions in order to be effectively solved, hence requiring a specific number (possibly large) of DGN layers.

Original: Despite the progress made in recent years in the field, many of the proposed methods suffer from the over-squashing problem (Alon & Yahav, 2021) when the number of layers increases.
Revised: Despite the progress made in recent years in the field, many of the proposed methods suffer from the over-squashing problem (Alon & Yahav, 2021) when the number of layers increases.

Original: Specifically, when increasing the number of layers to cater for longer-range interactions, one observes an excessive amplification or an annihilation of the information being routed to the node by the message passing process to update its fixed length encoding.
Revised: Specifically, when increasing the number of layers to cater for longer-range interactions, one observes an excessive amplification or an annihilation of the information being routed to the node by the message passing process to update its fixed length encoding.

Original: As such, over-squashing prevents DGNs to learn long-range information.
Revised: As such, over-squashing prevents DGNs to learn long-range information.

Original: In this work, we present Anti-Symmetric Deep Graph Network (A-DGN), a framework for effective long-term propagation of information in DGN architectures designed through the lens of ordinary differential equations (ODEs).
Revised: In this work, we present Anti-Symmetric Deep Graph Network (A-DGN), a framework for effective long-term propagation of information in DGN architectures designed through the lens of ordinary differential equations (ODEs).

Original: Leveraging the connections between ODEs and deep neural architectures, we provide theoretical conditions for realizing a stable and non-dissipative ODE system on graphs through the use of anti-symmetric weight matrices.
Revised: Leveraging the connections between ODEs and deep neural architectures, we provide theoretical conditions for realizing a stable and non-dissipative ODE system on graphs through the use of anti-symmetric weight matrices.

Original: The formulation of the A-DGN layer then results from the forward Euler discretization of the achieved graph ODE.
Revised: The formulation of the A-DGN layer then results from the forward Euler discretization of the achieved graph ODE.

Original: Thanks to the properties enforced on the ODE, our framework preserves the long-term dependencies between nodes as well as prevents from gradient explosion or vanishing.
Revised: Thanks to the properties enforced on the ODE, our framework preserves the long-term dependencies between nodes as well as prevents from gradient explosion or vanishing.

Original: Interestingly, our analysis also paves the way for rethinking the formulation of standard DGNs as discrete versions of non-dissipative and stable
Revised: Interestingly, our analysis also paves the way for rethinking the formulation of standard DGNs as discrete versions of non-dissipative and stable

Original: ODEs on graphs.
Revised: ODEs on graphs.

Original: The key contributions of this work can be summarized as follows:
Revised: The key contributions of this work can be summarized as follows:

Original: • We introduce A-DGN, a novel design scheme for deep graph networks stemming from an ODE formulation.
Revised: • We introduce A-DGN, a novel design scheme for deep graph networks stemming from an ODE formulation.

Original: Stability and non-dissipation are the main properties that characterize our method, allowing the preservation of long-term dependencies in the information flow.
Revised: Stability and non-dissipation are the main properties that characterize our method, allowing the preservation of long-term dependencies in the information flow.

Original: • We theoretically prove that the employed ODE on graphs has stable and non-dissipative behavior.
Revised: • We theoretically prove that the employed ODE on graphs has stable and non-dissipative behavior.

Original: Such result leads to the absence of exploding and vanishing gradient problems during training, typical of unstable and lossy systems.
Revised: Such result leads to the absence of exploding and vanishing gradient problems during training, typical of unstable and lossy systems.

Original: • We conduct extensive experiments to demonstrate the benefits of our method.
Revised: • We conduct extensive experiments to demonstrate the benefits of our method.

Original: A-DGN can outperform classical DGNs over several datasets even when dozens of layers are used.
Revised: A-DGN can outperform classical DGNs over several datasets even when dozens of layers are used.

Original: The rest of this paper is organized as follows.
Revised: The rest of this paper is organized as follows.

Original: We introduce the A-DGN framework in Sectionby theoretically proving its properties.
Revised: We introduce the A-DGN framework in Sectionby theoretically proving its properties.

Original: In Section 3, we give an overview of the related work in the field of representation learning for graphs and continuous dynamic models.
Revised: In Section 3, we give an overview of the related work in the field of representation learning for graphs and continuous dynamic models.

Original: Afterwards, we provide the experimental assessment of our method in Section 4.
Revised: Afterwards, we provide the experimental assessment of our method in Section 4.

Original: Finally, Section 5 concludes the paper.
Revised: Finally, Section 5 concludes the paper.

Original:  A NTI -S YMMETRIC D EEP G RAPH N ETWORK
Revised:  A NTI -S YMMETRIC D EEP G RAPH N ETWORK

Original: Recent advancements in the field of representation learning propose to treat neural network architectures as an ensemble of continuous (rather than discrete) layers, thereby drawing connections between deep neural networks and ordinary differential equations (ODEs) (Haber & Ruthotto, 2017;
Revised: Recent advancements in the field of representation learning propose to treat neural network architectures as an ensemble of continuous (rather than discrete) layers, thereby drawing connections between deep neural networks and ordinary differential equations (ODEs) (Haber & Ruthotto, 2017;

Original: Chen et al., 2018).
Revised: Chen et al., 2018).

Original: This connection can be pushed up to neural processing of graphs as introduced in (Poli et al., 2019), by making a suitable ODE define the computation on a graph structure.
Revised: This connection can be pushed up to neural processing of graphs as introduced in (Poli et al., 2019), by making a suitable ODE define the computation on a graph structure.

Original: We focus on static graphs, i.e., on structures described by G =
Revised: We focus on static graphs, i.e., on structures described by G =

Original: ( V , E ) , with V and E respectively denoting the fixed sets of nodes and edges.
Revised: ( V , E ) , with V and E respectively denoting the fixed sets of nodes and edges.

Original: For each node u ∈ V we consider a state x u ( t ) ∈ R d , which provides a representation of the node u at time t .
Revised: For each node u ∈ V we consider a state x u ( t ) ∈ R d , which provides a representation of the node u at time t .

Original: We can then define a Cauchy problem on graphs in terms of the following node-wise defined ODE:
Revised: We can then define a Cauchy problem on graphs in terms of the following node-wise defined ODE:

Original: [Equation]
Revised: [Equation]

Original: for time t ∈ [0 , T ] , and subject to the initial condition x u (0) =
Revised: for time t ∈ [0 , T ] , and subject to the initial condition x u (0) =

Original: x 0 u ∈ R d .
Revised: x 0 u ∈ R d .

Original: The dynamics of node’s representations is described by the function f G : R d → R d , while the initial condition x u (0) can be interpreted as the initial configuration of the node’s information, hence as the input for our computational model.
Revised: The dynamics of node’s representations is described by the function f G : R d → R d , while the initial condition x u (0) can be interpreted as the initial configuration of the node’s information, hence as the input for our computational model.

Original: As a consequence, the ODE defined in Equation 1 can be seen as a continuous information processing system over the graph, which starting from the input configuration x u (0) computes the final node’s representation (i.e., embedding) x u ( T ) .
Revised: As a consequence, the ODE defined in Equation 1 can be seen as a continuous information processing system over the graph, which starting from the input configuration x u (0) computes the final node’s representation (i.e., embedding) x u ( T ) .

Original: Notice that this process shares similarities with standard DGNs, in what it computes nodes’ states that can be used as an embedded representation of the graph and then used to feed a readout layer in a downstream task on graphs.
Revised: Notice that this process shares similarities with standard DGNs, in what it computes nodes’ states that can be used as an embedded representation of the graph and then used to feed a readout layer in a downstream task on graphs.

Original: The top of Figure 1 visually summarizes this concept, showing how nodes evolve following a specific graph ODE in the time span between 0 and a terminal time T > 0 .
Revised: The top of Figure 1 visually summarizes this concept, showing how nodes evolve following a specific graph ODE in the time span between 0 and a terminal time T > 0 .

Original: Since for most ODEs it is impractical to compute an analytical solution, a common approach relies on finding an approximate one through a numerical discretization procedure (such as the forward
Revised: Since for most ODEs it is impractical to compute an analytical solution, a common approach relies on finding an approximate one through a numerical discretization procedure (such as the forward

Original: Euler method).
Revised: Euler method).

Original: In this way, the time variable is discretized and the ODE solution is computed by the successive application of an iterated map that operates on the discrete set of points between 0 and T , with a step size ϵ > 0 .
Revised: In this way, the time variable is discretized and the ODE solution is computed by the successive application of an iterated map that operates on the discrete set of points between 0 and T , with a step size ϵ > 0 .

Original: Crucially, as already observed for feed-forward and recurrent neural models (Haber & Ruthotto, 2017; Chang et al., 2019), each step of the ODE discretization process can be equated to one layer of a DGN network.
Revised: Crucially, as already observed for feed-forward and recurrent neural models (Haber & Ruthotto, 2017; Chang et al., 2019), each step of the ODE discretization process can be equated to one layer of a DGN network.

Original: The whole neural architecture contains as many layers as the integration steps in the numerical method (i.e., L = T/ϵ ), and each layer ℓ = 1 , ..., L computes nodes’ states x ℓu which approximates x u ( ϵ ℓ ) .
Revised: The whole neural architecture contains as many layers as the integration steps in the numerical method (i.e., L = T/ϵ ), and each layer ℓ = 1 , ..., L computes nodes’ states x ℓu which approximates x u ( ϵ ℓ ) .

Original: This process is summarized visually in the middle of
Revised: This process is summarized visually in the middle of

Original: Figure 1.
Revised: Figure 1.

Original: Leveraging the concept of graph neural ODEs (Poli et al., 2019), in this paper we perform a further step by reformulating a DGN as a solution to a stable and non-dissipative Cauchy problem over a graph.
Revised: Leveraging the concept of graph neural ODEs (Poli et al., 2019), in this paper we perform a further step by reformulating a DGN as a solution to a stable and non-dissipative Cauchy problem over a graph.

Original: The main goal of our work is therefore achieving preservation of long-range information between nodes, while laying down the conditions that prevent gradient vanishing or explosion.
Revised: The main goal of our work is therefore achieving preservation of long-range information between nodes, while laying down the conditions that prevent gradient vanishing or explosion.

Original: Inspired by the works on stable deep architectures that discretize ODE solutions (Haber & Ruthotto, 2017; Chang et al., 2019), we do so by first deriving conditions under which the graph ODE is constrained to the desired stable and non-dissipative behavior.
Revised: Inspired by the works on stable deep architectures that discretize ODE solutions (Haber & Ruthotto, 2017; Chang et al., 2019), we do so by first deriving conditions under which the graph ODE is constrained to the desired stable and non-dissipative behavior.

Original: Since we are dealing with static graphs, we instantiate Equation 1 for a node u as follows:
Revised: Since we are dealing with static graphs, we instantiate Equation 1 for a node u as follows:

Original: [Equation]
Revised: [Equation]

Original: where σ is a monotonically non-decreasing activation function, W t ∈ R d × d and b t ∈ R d are, respectively, a weight matrix and a bias vector that contain the trainable parameters of the system.
Revised: where σ is a monotonically non-decreasing activation function, W t ∈ R d × d and b t ∈ R d are, respectively, a weight matrix and a bias vector that contain the trainable parameters of the system.

Original: We denote by Φ( X ( t ) , N u ) the aggregation function for the states of the nodes in the neighborhood of u .
Revised: We denote by Φ( X ( t ) , N u ) the aggregation function for the states of the nodes in the neighborhood of u .

Original: We refer to X ( t ) ∈ R |V|× d as the node feature matrix of the whole graph, with d the number of available features.
Revised: We refer to X ( t ) ∈ R |V|× d as the node feature matrix of the whole graph, with d the number of available features.

Original: For simplicity, in the following we keep W t and b t constant over time, hence dropping the t subscript in the notation.
Revised: For simplicity, in the following we keep W t and b t constant over time, hence dropping the t subscript in the notation.

Original: Well-posedness and stability are essential concepts when designing DGNs as solutions to Cauchy problems, both relying on the continuous dependence of the solution from initial conditions.
Revised: Well-posedness and stability are essential concepts when designing DGNs as solutions to Cauchy problems, both relying on the continuous dependence of the solution from initial conditions.

Original: An ill-posed unstable system, even if potentially yielding a low training error, is likely to lead to a poor generalization error on perturbed data.
Revised: An ill-posed unstable system, even if potentially yielding a low training error, is likely to lead to a poor generalization error on perturbed data.

Original: On the other hand, the solution of a Cauchy problem is stable if the long-term behavior of the system does not depend significantly on the initial conditions (Ascher et al., 1995).
Revised: On the other hand, the solution of a Cauchy problem is stable if the long-term behavior of the system does not depend significantly on the initial conditions (Ascher et al., 1995).

Original: In our case, where the ODE defines a message passing diffusion over a graph, our intuition is that a stable encoding system will be robust to perturbations in the input nodes information.
Revised: In our case, where the ODE defines a message passing diffusion over a graph, our intuition is that a stable encoding system will be robust to perturbations in the input nodes information.

Original: Hence, the state representations will change smoothly with the input, resulting in a non-exploding forward propagation and better generalization.
Revised: Hence, the state representations will change smoothly with the input, resulting in a non-exploding forward propagation and better generalization.

Original: This intuition is formalized by the
Revised: This intuition is formalized by the

Original: Definition 1 (see Appendix A), which idea is that a small perturbation of size δ of the initial state (i.e., the node input features) results in a perturbation on the subsequent states that is at most ω .
Revised: Definition 1 (see Appendix A), which idea is that a small perturbation of size δ of the initial state (i.e., the node input features) results in a perturbation on the subsequent states that is at most ω .

Original: As known from the stability theory of autonomous systems (Ascher et al., 1995), this condition is met when the maximum real part of the Jacobian’s eigenvalues of f G is smaller or equal than 0, i.e., max i =1 ,...,d Re ( λ i ( J ( t )))
Revised: As known from the stability theory of autonomous systems (Ascher et al., 1995), this condition is met when the maximum real part of the Jacobian’s eigenvalues of f G is smaller or equal than 0, i.e., max i =1 ,...,d Re ( λ i ( J ( t )))

Original: ≤ 0 , ∀ t ≥ 0 .
Revised: ≤ 0 , ∀ t ≥ 0 .

Original: Although stability is a necessary condition for successful learning, it alone is not sufficient to capture long-term dependencies.
Revised: Although stability is a necessary condition for successful learning, it alone is not sufficient to capture long-term dependencies.

Original: As it is discussed in Haber & Ruthotto (2017), if max i =1 ,...,d Re ( λ i ( J ( t )))
Revised: As it is discussed in Haber & Ruthotto (2017), if max i =1 ,...,d Re ( λ i ( J ( t )))

Original: ≪ 

Revised: ≪

Original: the result is a lossy system subject to catastrophic forgetting during propagation.
Revised: the result is a lossy system subject to catastrophic forgetting during propagation.

Original: Thus, in the graph domain, this means that only local neighborhood information is preserved by the system, while long-range dependencies among nodes are forgotten.
Revised: Thus, in the graph domain, this means that only local neighborhood information is preserved by the system, while long-range dependencies among nodes are forgotten.

Original: If no longrange information is preserved, then it is likely that the DGN will underperform, since it will not be able to reach the minimum radius of inter-nodes interactions needed to effectively solve the task.
Revised: If no longrange information is preserved, then it is likely that the DGN will underperform, since it will not be able to reach the minimum radius of inter-nodes interactions needed to effectively solve the task.

Original: Therefore, we can design an ODE for graphs which is stable and non-dissipative (see Definitionin Appendix A) that leads to well-posed learning, when the criterion that guarantees stability is met and the Jacobian’s eigenvalues of f G are nearly zero.
Revised: Therefore, we can design an ODE for graphs which is stable and non-dissipative (see Definitionin Appendix A) that leads to well-posed learning, when the criterion that guarantees stability is met and the Jacobian’s eigenvalues of f G are nearly zero.

Original: Under this condition, the forward propagation produces at most moderate amplification or shrinking of the input, which enables to preserve longterm dependencies in the node states.
Revised: Under this condition, the forward propagation produces at most moderate amplification or shrinking of the input, which enables to preserve longterm dependencies in the node states.

Original: During training, the backward propagation needed to compute the gradient of the loss ∂
Revised: During training, the backward propagation needed to compute the gradient of the loss ∂

Original: L /∂ x u ( t ) will have the same properties of the forward propagation.
Revised: L /∂ x u ( t ) will have the same properties of the forward propagation.

Original: As such, no gradient vanish nor explosion is expected to occur.
Revised: As such, no gradient vanish nor explosion is expected to occur.

Original: More formally:
Revised: More formally:

Original: Proposition 1.
Revised: Proposition 1.

Original: The forward and backward propagations of the ODE in Equation 2 are stable and non-dissipative if
Revised: The forward and backward propagations of the ODE in Equation 2 are stable and non-dissipative if

Original: [Equation]
Revised: [Equation]

Original: see the proof in Appendix B.
Revised: see the proof in Appendix B.

Original: A simple way to impose the condition in Equation 3 is to use an anti-symmetric 1 weight matrix in
Revised: A simple way to impose the condition in Equation 3 is to use an anti-symmetric 1 weight matrix in

Original: Equation 2.
Revised: Equation 2.

Original: Under this assumption, we can rewrite Equation 2 as follows:
Revised: Under this assumption, we can rewrite Equation 2 as follows:

Original: [Equation]
Revised: [Equation]

Original: where ( W − W T ) ∈ R
Revised: where ( W − W T ) ∈ R

Original: d × d is the anti-symmetric weight matrix.
Revised: d × d is the anti-symmetric weight matrix.

Original: As granted by the following
Revised: As granted by the following

Original: Proposition 2, the Jacobian of the resulting ODE has imaginary eigenvalues, hence it is stable and non-dissipative according to Proposition 1.
Revised: Proposition 2, the Jacobian of the resulting ODE has imaginary eigenvalues, hence it is stable and non-dissipative according to Proposition 1.

Original: Proposition 2.
Revised: Proposition 2.

Original: The Jacobian matrix of the ODE in Equation 4 has purely imaginary eigenvalues, i.e.
Revised: The Jacobian matrix of the ODE in Equation 4 has purely imaginary eigenvalues, i.e.

Original: [Equation]
Revised: [Equation]

Original: Therefore the ODE in Equation 4 is stable and non-dissipative.
Revised: Therefore the ODE in Equation 4 is stable and non-dissipative.

Original: See proof in Appendix C.
Revised: See proof in Appendix C.

Original: We now proceed to discretize the ODE in Equation 4 by means of the forward Euler’s method .
Revised: We now proceed to discretize the ODE in Equation 4 by means of the forward Euler’s method .

Original: To preserve stability of the discretized system 2 , we add a diffusion term to Equation 4, yielding the following node state update equation:
Revised: To preserve stability of the discretized system 2 , we add a diffusion term to Equation 4, yielding the following node state update equation:

Original: [Equation]
Revised: [Equation]

Original: where I is the identity matrix, γ is a hyper-parameter that regulates the strength of the diffusion, and ϵ is the discretization step.
Revised: where I is the identity matrix, γ is a hyper-parameter that regulates the strength of the diffusion, and ϵ is the discretization step.

Original: By building on the relationship between the discretization and the DGN layers, we have introduced x ℓu as the state of node u at layer ℓ , i.e. the discretization of state at time t = ϵℓ .
Revised: By building on the relationship between the discretization and the DGN layers, we have introduced x ℓu as the state of node u at layer ℓ , i.e. the discretization of state at time t = ϵℓ .

Original: Now, both ODE and its Euler discretization are stable and non-dissipative.
Revised: Now, both ODE and its Euler discretization are stable and non-dissipative.

Original: We refer to the framework defined by Equation 5 as Anti-symmetric Deep Graph Network (A-DGN), whose state update process is schematically illustrated in the bottom of Figure 1.
Revised: We refer to the framework defined by Equation 5 as Anti-symmetric Deep Graph Network (A-DGN), whose state update process is schematically illustrated in the bottom of Figure 1.

Original: Notice that having assumed the parameters of the ODE constant in time, A-DGN can also be interpreted as a recursive DGN with weight sharing between layers.
Revised: Notice that having assumed the parameters of the ODE constant in time, A-DGN can also be interpreted as a recursive DGN with weight sharing between layers.

Original: We recall that Φ( X ℓ − 1 , N u ) can be any function that aggregates nodes (and edges) information.
Revised: We recall that Φ( X ℓ − 1 , N u ) can be any function that aggregates nodes (and edges) information.

Original: Therefore, the general formulation of Φ( X ℓ − 1 , N u ) in A-DGN allows casting all standard DGNs through in their non-dissipative, stable and well-posed version.
Revised: Therefore, the general formulation of Φ( X ℓ − 1 , N u ) in A-DGN allows casting all standard DGNs through in their non-dissipative, stable and well-posed version.

Original: As a result, A-DGN can be implemented leveraging the aggregation function that is more adequate for the specific task, while allowing to preserve long-range relationships in the graph.
Revised: As a result, A-DGN can be implemented leveraging the aggregation function that is more adequate for the specific task, while allowing to preserve long-range relationships in the graph.

Original: As a demonstration of this, in Sectionwe explore two neighborhood aggregation functions, that are
Revised: As a demonstration of this, in Sectionwe explore two neighborhood aggregation functions, that are

Original: [Equation]
Revised: [Equation]

Original: (which is also employed in Morris et al.
Revised: (which is also employed in Morris et al.

Original: and the classical GCN aggregation
Revised: and the classical GCN aggregation

Original: [Equation]
Revised: [Equation]

Original: where V is the weight matrix, ˆ d j and ˆ d u are, respectively, the degrees of nodes j and u .
Revised: where V is the weight matrix, ˆ d j and ˆ d u are, respectively, the degrees of nodes j and u .

Original: Finally, although we designed A-DGN with weight sharing in mind (for ease of presentation), a more general version of the framework, with layer-dependent weights W ℓ − ( W ℓ ) T , is possible 3 .
Revised: Finally, although we designed A-DGN with weight sharing in mind (for ease of presentation), a more general version of the framework, with layer-dependent weights W ℓ − ( W ℓ ) T , is possible 3 .

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Deep Graph Network Nowadays, most of the DGNs typically relies on the concepts introduced by the Message Passing Neural Network (MPNN) (Gilmer et al., 2017), which is a general framework based on the message passing paradigm.
Revised: Deep Graph Network Nowadays, most of the DGNs typically relies on the concepts introduced by the Message Passing Neural Network (MPNN) (Gilmer et al., 2017), which is a general framework based on the message passing paradigm.

Original: The MPNN updates the representation for a node u at layer ℓ as
Revised: The MPNN updates the representation for a node u at layer ℓ as

Original: [Equation]
Revised: [Equation]

Original: where ϕ U and ϕ M are respectively the update and message functions.
Revised: where ϕ U and ϕ M are respectively the update and message functions.

Original: Hence, the role of the message function is to compute the message for each node, and then dispatch it among the neighbors.
Revised: Hence, the role of the message function is to compute the message for each node, and then dispatch it among the neighbors.

Original: On the other hand, the update function has the role of collecting the incoming messages and update the node state.
Revised: On the other hand, the update function has the role of collecting the incoming messages and update the node state.

Original: A typical implementation of the MPNN model is x
Revised: A typical implementation of the MPNN model is x

Original: [Equation]
Revised: [Equation]

Original: u
Revised: u

Original: [Equation]
Revised: [Equation]

Original: where e uv ∈ R
Revised: where e uv ∈ R

Original: d e is the edge feature vector between node u and v .
Revised: d e is the edge feature vector between node u and v .

Original: Thus, by relaxing the concepts of stability and non-dissipation from our framework, MPNN becomes a specific discretization instance of A-DGN.
Revised: Thus, by relaxing the concepts of stability and non-dissipation from our framework, MPNN becomes a specific discretization instance of A-DGN.

Original: Depending on the definition of the update and message functions, it is possible to derive a variety of DGNs that mainly differ on the neighbor aggregation scheme (Kipf & Welling, 2017; Veliˇckovi´c et al., 2018; Hamilton et al., 2017; Xu et al., 2019; Defferrard et al., 2016;
Revised: Depending on the definition of the update and message functions, it is possible to derive a variety of DGNs that mainly differ on the neighbor aggregation scheme (Kipf & Welling, 2017; Veliˇckovi´c et al., 2018; Hamilton et al., 2017; Xu et al., 2019; Defferrard et al., 2016;

Original: Hu et al., 2020).
Revised: Hu et al., 2020).

Original: However, all these methods focus on presenting new and effective functions without questioning the stability and non-dissipative behavior of the final network.
Revised: However, all these methods focus on presenting new and effective functions without questioning the stability and non-dissipative behavior of the final network.

Original: As a result, most of these DGNs are usually not able to capture long-term interactions.
Revised: As a result, most of these DGNs are usually not able to capture long-term interactions.

Original: Indeed, only few layers can be employed without falling into the over-squashing phenomenon, as it is discussed by Alon & Yahav (2021).
Revised: Indeed, only few layers can be employed without falling into the over-squashing phenomenon, as it is discussed by Alon & Yahav (2021).

Original: Since the previous methods are all specific cases of MPNN, they are all instances of the discretized and unconstrained version of A-DGN.
Revised: Since the previous methods are all specific cases of MPNN, they are all instances of the discretized and unconstrained version of A-DGN.

Original: Moreover, a proper design of Φ( X ℓ − 1 , N u ) in A-DGN allows rethinking the discussed DGNs through the lens of non-dissipative and stable ODEs.
Revised: Moreover, a proper design of Φ( X [Equation] − 1 , N u ) in A-DGN allows rethinking the discussed DGNs through the lens of non-dissipative and stable ODEs.

Original: Ming Chen properties discussed in this section are in fact still true even in the case of time varying W t in Equation 2, provided that max i =1 ,...,d Re ( λ i ( J ( t ))) ≤ 0 and J ( t ) change sufficiently slow over time (see (Ascher et al., 1995; Haber & Ruthotto, 2017)).
Revised: Ming Chen properties discussed in this section are in fact still true even in the case of time varying W t in Equation 2, provided that max i =1 ,...,d Re ( λ i ( J ( t ))) ≤ 0 and J ( t ) change sufficiently slow over time (see (Ascher et al., 1995; Haber & Ruthotto, 2017)).

Original: et al.
Revised: et al.

Original: (2020), Zhou et al.
Revised: (2020), Zhou et al.

Original: (2021), and Eliasof et al.
Revised: (2021), and Eliasof et al.

Original: (2022) proposed three methods to alleviate oversmoothing, which is a phenomenon where all node features become almost indistinguishable after few embedding updates.
Revised: (2022) proposed three methods to alleviate oversmoothing, which is a phenomenon where all node features become almost indistinguishable after few embedding updates.

Original: Similarly to the forward Euler discretization, the first method employs identity mapping.
Revised: Similarly to the forward Euler discretization, the first method employs identity mapping.

Original: It also exploits initial residual connections to ensure that the final representation of each node retains at least a fraction of input.
Revised: It also exploits initial residual connections to ensure that the final representation of each node retains at least a fraction of input.

Original: The second method proposes a DGN that constrains the Dirichlet energy at each layer and leverages initial residual connections, while the latter tackles over-smoothing by aggregating random paths over the graph nodes.
Revised: The second method proposes a DGN that constrains the Dirichlet energy at each layer and leverages initial residual connections, while the latter tackles over-smoothing by aggregating random paths over the graph nodes.

Revised: Thus, the novelty of our method is still preserved since A-DGN defines a map between DGNs and stable and non-dissipative graph

Revised: ODEs to preserve long-range dependencies between nodes.

Original: Continuous Dynamic Models Chen et al.
Revised: Continuous Dynamic Models Chen et al.

Original: (2018) introduce NeuralODE, a new family of neural network models that parametrizes the continuous dynamics of recurrent neural networks using ordinary differential equations.
Revised: (2018) introduce NeuralODE, a new family of neural network models that parametrizes the continuous dynamics of recurrent neural networks using ordinary differential equations.

Original: Similarly, Antisymmetric RNN (Chang et al., 2019) and Euler State
Revised: Similarly, Chang et al.

Original: Networks (Gallicchio, 2022) draw a connection between ODEs and, respectively, RNNs and Reservoir Computing architectures.
Revised: and Gallicchio (2022) draw a connection between ODEs and, respectively, RNNs and Reservoir Computing architectures.

Original: Both methods focus on the stability of the solution and the employed numerical method.
Revised: Both methods focus on the stability of the solution and the employed numerical method.

Original: Inspired by the NeuralODE approach, Poli et al.
Revised: Inspired by the NeuralODE approach, Poli et al.

Original: (2019) develops a DGN defined as a continuum of layers.
Revised: (2019) develops a DGN defined as a continuum of layers.

Original: In such a work, the authors focus on building the connection between ODEs and DGNs.
Revised: In such a work, the authors focus on building the connection between ODEs and DGNs.

Original: We extend their work to include stability and non-dissipation, which are fundamental properties to preserve long-term dependencies between nodes and prevent gradient explosion or vanishing during training.
Revised: We extend their work to include stability and non-dissipation, which are fundamental properties to preserve long-term dependencies between nodes and prevent gradient explosion or vanishing during training.

Original: Thus, by relaxing these two properties from our framework, the work by Poli et al.
Revised: Thus, by relaxing these two properties from our framework, the work by Poli et al.

Original: (2019) becomes a specific instance of A-DGN.
Revised: (2019) becomes a specific instance of A-DGN.

Original: Chamberlain et al.
Revised: Chamberlain et al.

Original: (2021) propose GRAND an architecture to learn graph diffusion as a partial derivative equation (PDE).
Revised: (2021) propose GRAND an architecture to learn graph diffusion as a partial derivative equation (PDE).

Original: Differently from
Revised: Differently from

Original: GRAND, our framework designs an architecture that is theoretically non-dissipative and free from gradient vanishing or explosion.
Revised: GRAND, our framework designs an architecture that is theoretically non-dissipative and free from gradient vanishing or explosion.

Original: DGC (Wang et al., 2021) and SGC (Wu et al., 2019) propose linear models that propagate node information as the discretization of the graph heat equation, ∂ X ( t ) /∂t
Revised: DGC (Wang et al., 2021) and SGC (Wu et al., 2019) propose linear models that propagate node information as the discretization of the graph heat equation, ∂ X ( t ) /∂t

Original: = − LX ( t ) , without learning.
Revised: = − LX ( t ) , without learning.

Original: Specifically, DGC mainly focus on exploring the influence of the step size ϵ in the Euler discretization method.
Revised: Specifically, DGC mainly focus on exploring the influence of the step size ϵ in the Euler discretization method.

Original: Eliasof et al.
Revised: Eliasof et al.

Original: (2021) and Rusch et al.
Revised: (2021) and Rusch et al.

Original: (2022) present two methods to preserve the energy of the system, i.e., they mitigate over-smoothing, instead of preserving long-range information between nodes.
Revised: (2022) present two methods to preserve the energy of the system, i.e., they mitigate over-smoothing, instead of preserving long-range information between nodes.

Original: Differently from our method, which employs a first-order ODE, the former leverages the conservative mapping defined by hyperbolic PDEs, while the latter is defined as second-order ODEs that preserve the Dirichlet energy.
Revised: Differently from our method, which employs a first-order ODE, the former leverages the conservative mapping defined by hyperbolic PDEs, while the latter is defined as second-order ODEs that preserve the Dirichlet energy.

Original: In general, this testifies that non-dissipation in graph ODEs is an important property to pursue, not only when preserving long-range dependencies.
Revised: In general, this testifies that non-dissipation in graph ODEs is an important property to pursue, not only when preserving long-range dependencies.

Original: However, to the best of our knowledge, we are the first to propose a non-dissipative graph ODE to effectively propagate the information on the graph structure.
Revised: However, to the best of our knowledge, we are the first to propose a non-dissipative graph ODE to effectively propagate the information on the graph structure.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we discuss the empirical assessment of our method.
Revised: In this section, we discuss the empirical assessment of our method.

Original: Specifically,in Section 4.1, we evaluate our framework ona graph property prediction task where we predict single source shortest path, node eccentricity, and graph diameter.
Revised: Specifically, we show the efficacy of preserving long-range information between nodes and mitigating the over-squashing by evaluating our framework on graph property prediction tasks where we predict single source shortest path, node eccentricity, and graph diameter (see Section 4.1).

Original: In Section 4.2, we evaluate A-DGN  onfive classical graph  benchmarks.
Revised: With the same purpose, we report the experiments on the TreeNeighboursMatch problem (Alon & Yahav, 2021) in Appendix G.2. Moreover, we assess the performance of the proposed A-DGN approach on classical graph homophilic (see Section 4.2) and heterophilic (see Appendix G.1) benchmarks.

Original: The performance of A-DGN is assessed against DGN variants from the literature.
Revised: The performance of A-DGN is assessed against DGN variants from the literature.

Original: We report additional experiments on graph heterophilic datasets in Appendix G.

Original: We carried the experiments on a Dell server with 4 Nvidia GPUs A100.
Revised: We carried the experiments on a Dell server with 4 Nvidia GPUs A100.

Original: We release openly the code implementing our methodology and reproducing our empirical analysis after acceptance.
Revised: We release openly the code implementing our methodology and reproducing our empirical analysis after acceptance.

Original: 4. G RAPH PROPERTY PREDICTION
Revised: 4. G RAPH PROPERTY PREDICTION

Original: Setup For the graph property prediction task, we considered three datasets extracted from the work of Corso et al.
Revised: Setup For the graph property prediction task, we considered three datasets extracted from the work of Corso et al.

Original: The analysis consists of classical graph theory tasks on undirected unweighted randomly generated graphs sampled from a wide variety of distributions.
Revised: The analysis consists of classical graph theory tasks on undirected unweighted randomly generated graphs sampled from a wide variety of distributions.

Original: Specifically, we considered two node level tasks and one graph level task, which are single source shortest path (SSSP), node eccentricity, and graph diameter.
Revised: Specifically, we considered two node level tasks and one graph level task, which are single source shortest path (SSSP), node eccentricity, and graph diameter.

Original: Such tasks require capturing long-term dependencies in order to be solved, thus mitigating the over-squashing phenomenon.
Revised: Such tasks require capturing long-term dependencies in order to be solved, thus mitigating the over-squashing phenomenon.

Original: Indeed, in the SSSP task, we are computing the shortest paths between a given node u and all other nodes in the graph.
Revised: Indeed, in the SSSP task, we are computing the shortest paths between a given node u and all other nodes in the graph.

Original: Thus, it is fundamental to propagate not only the information of the direct neighborhood of u , but also the information of nodes which are extremely far from it.
Revised: Thus, it is fundamental to propagate not only the information of the direct neighborhood of u , but also the information of nodes which are extremely far from it.

Original: Similarly, for diameter and eccentricity.
Revised: Similarly, for diameter and eccentricity.

Original: We employed the same seed and generator as Corso et al.
Revised: We employed the same seed and generator as Corso et al.

Original: (2020) to generate the datasets,
Revised: (2020) to generate the datasets,

Original: but we considered graphs with 25 to 35 nodes, instead of 15-25 nodes as in the original work, to increase the task complexity and lengthen long-range dependencies required to solve the task.
Revised: but we considered graphs with 25 to 35 nodes, instead of 15-25 nodes as in the original work, to increase the task complexity and lengthen long-range dependencies required to solve the task.

Original: As in the original work, we used 5120 graphs as training set, 640 as validation set, and 1280 as test set.
Revised: As in the original work, we used 5120 graphs as training set, 640 as validation set, and 1280 as test set.

Original: We explored the performance of two versions of A-DGN, i.e., weight sharing and layer dependent weights.
Revised: We explored the performance of two versions of A-DGN, i.e., weight sharing and layer dependent weights.

Original: Moreover, we employed two instances of our method leveraging the two aggregation functions in Equation 6 and 7.
Revised: Moreover, we employed two instances of our method leveraging the two aggregation functions in Equation 6 and 7.

Original: We will refer to the former as simple aggregation and to the latter as
Revised: We will refer to the former as simple aggregation and to the latter as

Original: GCN-based aggregation.
Revised: GCN-based aggregation.

Original: We compared our method with GCNII, two neuralODE-based models, i.e., GRAND and DGC, and the four most popular DGNs, i.e., GCN, GAT, GraphSAGE, and GIN.
Revised: We compared our method with GCNII, two neuralODE-based models, i.e., GRAND and DGC, and the four most popular DGNs, i.e., GCN, GAT, GraphSAGE, and GIN.

Original: We designed each model as a combination of three main components.
Revised: We designed each model as a combination of three main components.

Original: The first is the encoder which maps the node input features into a latent hidden space; the second is the graph convolution (i.e.,
Revised: The first is the encoder which maps the node input features into a latent hidden space; the second is the graph convolution (i.e.,

Original: A-DGN or the DGN baseline); and the third is a readout that maps the output of the convolution into the output space.
Revised: A-DGN or the DGN baseline); and the third is a readout that maps the output of the convolution into the output space.

Original: The encoder and the readout are Multi-Layer Perceptrons that share the same architecture among all models in the experiments.
Revised: The encoder and the readout are Multi-Layer Perceptrons that share the same architecture among all models in the experiments.

Original: We performed hyper-parameter tuning via grid search, optimizing the Mean Square Error (MSE).
Revised: We performed hyper-parameter tuning via grid search, optimizing the Mean Square Error (MSE).

Original: We trained the models using Adam optimizer for a maximum of 1500 epochs and early stopping with patience of 100 epochs on the validation error.
Revised: We trained the models using Adam optimizer for a maximum of 1500 epochs and early stopping with patience of 100 epochs on the validation error.

Original: For each model configuration, we performedtraining runs with different weight initialization and report the average of the results.
Revised: For each model configuration, we performedtraining runs with different weight initialization and report the average of the results.

Original: We report in Appendix E the grid of hyper-parameters exploited for this experiment.
Revised: We report in Appendix E the grid of hyper-parameters exploited for this experiment.

Original: We observe that even if we do not directly explore in the hyper-parameter space the terminal time T in which the node evolution produces the best embeddings, that is done indirectly by fixing the values of the step size ϵ and the maximum number of layers L , since T
Revised: We observe that even if we do not directly explore in the hyper-parameter space the terminal time T in which the node evolution produces the best embeddings, that is done indirectly by fixing the values of the step size ϵ and the maximum number of layers L , since T

Original: =
Revised: =

Original: Lϵ .
Revised: Lϵ .

Original:  present the results on the graph property prediction in Table 1.
Revised: Results We present the results on the graph property prediction in Table 1.

Original: Specifically, we report log 10 (MSE) as the evaluation metric.
Revised: Specifically, we report log 10 (MSE) as the evaluation metric.

Original: We observe that our method, A-DGN, outperforms all the
Revised: We observe that our method, A-DGN, outperforms all the

Original: DGNs employed in this experiment.
Revised: DGNs employed in this experiment.

Original: Indeed, by employing GCN-based aggregation, we achieve an error score that is on average 0.80 points better than the selected baselines.
Revised: Indeed, by employing GCN-based aggregation, we achieve an error score that is on average 0.80 points better than the selected baselines.

Original: Notably, if we leverage the simple aggregation scheme, A-DGN increases the performance gap from the baselines.
Revised: Notably, if we leverage the simple aggregation scheme, A-DGN increases the performance gap from the baselines.

Original: A-DGN with simple aggregation shows a decisive improvement with respect to baselines.
Revised: A-DGN with simple aggregation shows a decisive improvement with respect to baselines.

Original: Specifically, it achieves a performance that is 200% to 300% better than the best baseline in each task.
Revised: Specifically, it achieves a performance that is 200% to 300% better than the best baseline in each task.

Original: Moreover, it is on average 3 . 6 × faster that the baselines (see Table 7 in Appendix F).
Revised: Moreover, it is on average 3 . 6 × faster that the baselines (see Table 7 in Appendix F).

Original: We observe that the main challenge when predicting diameter, eccentricity, or SSSP is to leverage not only local information but also global graph information.
Revised: We observe that the main challenge when predicting diameter, eccentricity, or SSSP is to leverage not only local information but also global graph information.

Original: Such knowledge can only be learned by exploring long-range dependencies.
Revised: Such knowledge can only be learned by exploring long-range dependencies.

Original: Indeed, the three tasks are extremely correlated.
Revised: Indeed, the three tasks are extremely correlated.

Original: All of them require to compute shortest paths in the graph.
Revised: All of them require to compute shortest paths in the graph.

Original: Thus, as for standard algorithmic solutions (e.g., Bellman–Ford (Bellman, 1958), Dijkstra’s algorithm (Dijkstra, 1959)), more messages between nodes need to be exchanged in order to achieve accurate solutions.
Revised: Thus, as for standard algorithmic solutions (e.g., Bellman–Ford (Bellman, 1958), Dijkstra’s algorithm (Dijkstra, 1959)), more messages between nodes need to be exchanged in order to achieve accurate solutions.

Original: This suggests that A-DGN can better capture and exploit such information.
Revised: This suggests that A-DGN can better capture and exploit such information.

Original: Moreover, this indicates also that the simple aggregator is more effective than the GCN-based because the tasks are mainly based on counting distances.
Revised: Moreover, this indicates also that the simple aggregator is more effective than the GCN-based because the tasks are mainly based on counting distances.

Original: Thus, exploiting the information derived from the Laplacian operator is not helpful for solving this kind of algorithmic tasks.
Revised: Thus, exploiting the information derived from the Laplacian operator is not helpful for solving this kind of algorithmic tasks.

Original: 4. G RAPH BENCHMARKS
Revised: 4. G RAPH BENCHMARKS

Original: Setup In the graph benchmark setting we consider five well-known graph datasets for node classification, i.e., PubMed (Namata et al., 2012); coauthor graphs CS and Physics; and the Amazon co-purchasing graphs Computer and Photo from Shchur et al.
Revised: Setup In the graph benchmark setting we consider five well-known graph datasets for node classification, i.e., PubMed (Namata et al., 2012); coauthor graphs CS and Physics; and the Amazon co-purchasing graphs Computer and Photo from Shchur et al.

Original: Also for this class of experiments, we considered the same baselines and architectural choices as for the graph property prediction task.
Revised: Also for this class of experiments, we considered the same baselines and architectural choices as for the graph property prediction task.

Original: However, in this experiment we study only the version of A-DGN with weight sharing, since it achieve good performances with low training costs.
Revised: However, in this experiment we study only the version of A-DGN with weight sharing, since it achieve good performances with low training costs.

Original: Within the aim to accurately assess the generalization performance of the models, we randomly split the datasets into multiple train/validation/test sets.
Revised: Within the aim to accurately assess the generalization performance of the models, we randomly split the datasets into multiple train/validation/test sets.

Original: Similarly to Shchur et al.
Revised: Similarly to Shchur et al.

Original: (2018), we use 20 labeled nodes per class as the training set, 30 nodes per class as the validation set, and the rest  as the test set.
Revised: (2018), we use 20 labeled nodes per class as the training set, 30 nodes per class as the validation set, and the rest [Table] as the test set.

Original: We generate 5 random splits per dataset and 5 random weight initialization for each configuration in each split.
Revised: We generate 5 random splits per dataset and 5 random weight initialization for each configuration in each split

Original: [Table]
Revised: [Table]

Original: We perform hyper-parameter tuning via grid search, optimizing the accuracy score.
Revised: We perform hyper-parameter tuning via grid search, optimizing the accuracy score.

Original: We train for a maximum of 10000 epochs to minimize the Cross-Entropy loss.
Revised: We train for a maximum of 10000 epochs to minimize the Cross-Entropy loss.

Original: We use an early stopping criterion that stops the training if the validation score does not improve for 50 epochs.
Revised: We use an early stopping criterion that stops the training if the validation score does not improve for 50 epochs.

Original: We report in
Revised: We report in

Original: Appendix E the grid of hyper-parameters explored for this experiment.
Revised: Appendix E the grid of hyper-parameters explored for this experiment.

Original: Results We present the results on the graph benchmark in Table 2.
Revised: Results We present the results on the graph benchmark in Table 2.

Original: Specifically, we report the accuracy as the evaluation metric.
Revised: Specifically, we report the accuracy as the evaluation metric.

Original: Even in this scenario, A-DGN outperforms the selected baselines, except in PubMed and Amazon Computers where GCNII is slightly better than our method.
Revised: Even in this scenario, A-DGN outperforms the selected baselines, except in PubMed and Amazon Computers where GCNII is slightly better than our method.

Original: In this benchmark, results that the GCN-based aggregation produces higher scores with respect to the simple aggregation.
Revised: In this benchmark, results that the GCN-based aggregation produces higher scores with respect to the simple aggregation.

Original: Thus, additional local neighborhood features extracted from the graph Laplacian seem to strengthen the final predictions.
Revised: Thus, additional local neighborhood features extracted from the graph Laplacian seem to strengthen the final predictions.

Original: It appears also that there is less benefit from including global information with respect to the graph property prediction scenario.
Revised: It appears also that there is less benefit from including global information with respect to the graph property prediction scenario.

Original: As a result, exploiting extremely long-range dependencies do not strongly improve the performance as the number of layers increases.
Revised: As a result, exploiting extremely long-range dependencies do not strongly improve the performance as the number of layers increases.

Original: To demonstrate that our approach performs well with many layers, we show in Figure 2 how the number of layers affects the accuracy score.
Revised: To demonstrate that our approach performs well with many layers, we show in Figure 2 how the number of layers affects the accuracy score.

Original: Our model maintains or improves the performance as the number of layers increases.
Revised: Our model maintains or improves the performance as the number of layers increases.

Original: On the other hand, all the baselines obtain good scores only with one to two layers and most of them exhibit a strong performance degradation as the number of layers increases.
Revised: On the other hand, all the baselines obtain good scores only with one to two layers and most of them exhibit a strong performance degradation as the number of layers increases.

Original: Indeed, in the Coauthor CS dataset we obtain that GraphSAGE, GAT, GCN and GIN lose 24.5% to 78.2% of accuracy.
Revised: Indeed, in the Coauthor CS dataset we obtain that GraphSAGE, GAT, GCN and GIN lose 24.5% to 78.2% of accuracy.

Original: We observe that DGC does not degrade its performance since the convolution do not contain parameters.
Revised: We observe that DGC does not degrade its performance since the convolution do not contain parameters.

Original: Although extreme long-range dependencies do not produce the same boost as in the graph property prediction scenario, including more than 5-hop neighborhoods is fundamental to improve stateof-the-art performances.
Revised: Although extreme long-range dependencies do not produce the same boost as in the graph property prediction scenario, including more than 5-hop neighborhoods is fundamental to improve stateof-the-art performances.

Original: As clear from Figure 2, this is not practical when standard DGNs are employed.
Revised: As clear from Figure 2, this is not practical when standard DGNs are employed.

Original: On the other hand, A-DGN demonstrates that can capture and exploit such information without any performance drop.
Revised: On the other hand, A-DGN demonstrates that can capture and exploit such information without any performance drop.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have presented Anti-Symmetric Deep Graph Network (A-DGN), a new framework for DGNs achieved from the study of ordinary differential equation (ODE) representing a continuous process of information diffusion on graphs.
Revised: In this paper, we have presented Anti-Symmetric Deep Graph Network (A-DGN), a new framework for DGNs achieved from the study of ordinary differential equation (ODE) representing a continuous process of information diffusion on graphs.

Original: Unlike previous approaches, by imposing stability and conservative constraints to the ODE through the use of anti-symmetric weight matrices, the proposed framework is able to learn and preserve long-range dependencies between nodes.
Revised: Unlike previous approaches, by imposing stability and conservative constraints to the ODE through the use of anti-symmetric weight matrices, the proposed framework is able to learn and preserve long-range dependencies between nodes.

Original: The ADGN layer formulation is the result of the forward Euler discretization of the corresponding Cauchy problem on graphs.
Revised: The ADGN layer formulation is the result of the forward Euler discretization of the corresponding Cauchy problem on graphs.

Original: We theoretically prove that the differential equation corresponding to A-DGN is stable as well as non-dissipative.
Revised: We theoretically prove that the differential equation corresponding to A-DGN is stable as well as non-dissipative.

Original: Consequently, typical problems of systems with unstable and lossydynamics, e.g., no gradient explosion or vanishing, do not occur.
Revised: Consequently, typical problems of systems with unstable and lossy

Original: Thanks to its formulation, A-DGN can be used to reinterpret and extend any classical DGN as a non-dissipative and stable ODEs.

Original: [Table]
Revised: [Table]

Original: Coauthor Physics, Amazon Computers, and Amazon Photo.
Revised: dynamics, e.g., no gradient explosion or vanishing, do not occur.

Original: The accuracy is averaged over 5 random train/validation/test splits and 5 random weight initialization of the best configuration per split.
Revised: Thanks to its formulation, A-DGN can be used to reinterpret and extend any classical DGN as a non-dissipative and stable ODEs.

Original: Our experimental analysis, on the one hand, shows that when capturing long-range dependencies is important for the task, our framework largely outperform standard DGNs.
Revised: Our experimental analysis, on the one hand, shows that when capturing long-range dependencies is important for the task, our framework largely outperform standard DGNs.

Original: On the other hand, it indicates the general competitiveness of A-DGN on several graph benchmarks.
Revised: On the other hand, it indicates the general competitiveness of A-DGN on several graph benchmarks.

Original: Overall, A-DGN shows the ability to effectively explore long-range dependencies and leverage dozens of layers without any noticeable drop in performance.
Revised: Overall, A-DGN shows the ability to effectively explore long-range dependencies and leverage dozens of layers without any noticeable drop in performance.

Original: For such reasons, we believe it can be a step towards the mitigation of the over-squashing problem in DGNs.
Revised: For such reasons, we believe it can be a step towards the mitigation of the over-squashing problem in DGNs.

Original: The results of our experimental analysis and the comparison with state-of-the-art methods suggest that the presented approach can be a starting point to mitigate also over-smoothing.
Revised: The results of our experimental analysis and the comparison with state-of-the-art methods suggest that the presented approach can be a starting point to mitigate also over-smoothing.

Original: Thus, we plan to accurately explore the consequences of our approach on over-smoothing as a future research direction.
Revised: Thus, we plan to accurately explore the consequences of our approach on over-smoothing as a future research direction.

Original: Looking ahead to other future developments, we plan to extend the analysis presented in this paper to study DGN architectures resulting from alternative discretization methods of the underlying graph ODE, e.g., using adaptive multi-step schemes (Ascher & Petzold, 1998).
Revised: Looking ahead to other future developments, we plan to extend the analysis presented in this paper to study DGN architectures resulting from alternative discretization methods of the underlying graph ODE, e.g., using adaptive multi-step schemes (Ascher & Petzold, 1998).

Original: Other future lines of investigation include extending the framework to dynamical graph structures, and evaluating its impact in the area of Reservoir Computing (Tanaka et al., 2019).
Revised: Other future lines of investigation include extending the framework to dynamical graph structures, and evaluating its impact in the area of Reservoir Computing (Tanaka et al., 2019).

Original: A S TABILITY AND DISSIPATIVITY DEFINITIONS
Revised: A S TABILITY AND DISSIPATIVITY DEFINITIONS

Original: In the following we define the condition in which the solution of the Cauchy problem in Equationis stable.
Revised: In the following we define the condition in which the solution of the Cauchy problem in Equationis stable.

Original: Definition 1.
Revised: Definition 1.

Original: A solution x u ( t ) of the ODE in Equation 2, with initial condition x u (0) , is stable if for any ω > 0 , there exists a δ > 0 such that any other solution ˜x u ( t ) of the ODE with initial condition ˜x u (0) satisfying | x u (0) − ˜x u (0)
Revised: A solution x u ( t ) of the ODE in Equation 2, with initial condition x u (0) , is stable if for any ω > 0 , there exists a δ > 0 such that any other solution ˜x u ( t ) of the ODE with initial condition ˜x u (0) satisfying | x u (0) − ˜x u (0)

Original: | ≤ δ also satisfies | x u ( t ) − ˜x u ( t ) | ≤ ω , for all t ≥ 0 .
Revised: | ≤ δ also satisfies | x u ( t ) − ˜x u ( t ) | ≤ ω , for all t ≥ 0 .

Original: We now provide a definition of a dissipative system based on the one provided in Humphries &
Revised: We now provide a definition of a dissipative system based on the one provided in Humphries &

Original: Stuart (1994).
Revised: Stuart (1994).

Original: Definition 2.
Revised: Definition 2.

Original: Let define E ⊆ R d a bounded set that contains any initial condition x u (0) for the ODE in Equation 2.
Revised: Let define E ⊆ R d a bounded set that contains any initial condition x u (0) for the ODE in Equation 2.

Original: The system defined by the ODE in Equation 2 is dissipative if there is a bounded set
Revised: The system defined by the ODE in Equation 2 is dissipative if there is a bounded set

Original: B where, for any E , exists t ∗ ≥ 0 such that { x u ( t ) | x u (0) ∈ E } ⊆ B for t > t ∗ .
Revised: B where, for any E , exists t ∗ ≥ 0 such that { x u ( t ) | x u (0) ∈ E } ⊆ B for t > t ∗ .

Original: B P ROOF OF P ROPOSITION
Revised: B P ROOF OF P ROPOSITION

Original: Let us consider the ODE defined in Equation 2 and analyze the sensitivity of its solution to the initial conditions.
Revised: Let us consider the ODE defined in Equation 2 and analyze the sensitivity of its solution to the initial conditions.

Original: Following (Chang et al., 2019), we differentiate both sides of Equation 2 with respect to x u (0) , obtaining:
Revised: Following (Chang et al., 2019), we differentiate both sides of Equation 2 with respect to x u (0) , obtaining:

Original: [Equation]
Revised: [Equation]

Original: Assuming the Jacobian does not change significantly over time, we can apply results from autonomous differential equations (Glendinning, 1994) and solve Equation 9 analytically as follows:
Revised: Assuming the Jacobian does not change significantly over time, we can apply results from autonomous differential equations (Glendinning, 1994) and solve Equation 9 analytically as follows:

Original: [Equation]
Revised: [Equation]

Original: where Λ is the diagonal matrix whose non-zero entries contain the eigenvalues of J , and T has the eigenvectors of J as columns.
Revised: where Λ is the diagonal matrix whose non-zero entries contain the eigenvalues of J , and T has the eigenvectors of J as columns.

Original: The qualitative behavior of ∂ x ( t ) /∂
Revised: The qualitative behavior of ∂ x ( t ) /∂

Original: x (0) is then determined by the real parts of the eigenvalues of J .
Revised: x (0) is then determined by the real parts of the eigenvalues of J .

Original: When max i =1 ,...,d Re ( λ i ( J ( t ))) > 0 , a small perturbation of the initial condition (i.e., a perturbation on the input graph) would cause an exponentially exploding difference in the nodes representations, and the system would be unstable.
Revised: When max i =1 ,...,d Re ( λ i ( J ( t ))) > 0 , a small perturbation of the initial condition (i.e., a perturbation on the input graph) would cause an exponentially exploding difference in the nodes representations, and the system would be unstable.

Original: On the contrary, for max i =1 ,...,d Re ( λ i ( J ( t )))
Revised: On the contrary, for max i =1 ,...,d Re ( λ i ( J ( t )))

Original: ≪ 0 input perturbations would rapidly vanish, hence the nodes’ representations would be insensitive to differences in the input graph, and the system would be dissipative.
Revised: ≪ 0 input perturbations would rapidly vanish, hence the nodes’ representations would be insensitive to differences in the input graph, and the system would be dissipative.

Original: If Re ( λ i ( J ( t )))
Revised: If Re ( λ i ( J ( t )))

Original: = 

Revised: =

Original: for i = 1 , ..., d then the magnitude of ∂ x ( t ) /∂
Revised: for i = 1 , ..., d then the magnitude of ∂ x ( t ) /∂

Original: x (0) is approximately constant over time, and the input graph information is effectively propagated through the successive transformations into the final nodes’ representations, with at most moderate amplification or loss.
Revised: x (0) is approximately constant over time, and the input graph information is effectively propagated through the successive transformations into the final nodes’ representations, with at most moderate amplification or loss.

Original: In this last case, the system is hence both stable and non-dissipative.
Revised: In this last case, the system is hence both stable and non-dissipative.

Original: Let us now consider a loss function L , and observe that its sensitivity to the initial condition (i.e., the input graph)
Revised: Let us now consider a loss function L , and observe that its sensitivity to the initial condition (i.e., the input graph)

Original: ∂
Revised: ∂

Original: L /∂ x u (0) is proportional to ∂ x u ( t ) /∂
Revised: L /∂ x u (0) is proportional to ∂ x u ( t ) /∂

Original: x u (0) .
Revised: x u (0) .

Original: Hence, in light of the previous considerations, if Re ( λ i ( J ( t ))) = 

Revised: Hence, in light of the previous considerations, if Re ( λ i ( J ( t ))) =

Original: for i = 1 , ..., d , then the magnitude of ∂ L /∂ x u (0) , which is the longest gradient chain that we can obtain during back-propagation, stays constant over time.
Revised: for i = 1 , ..., d , then the magnitude of ∂ L /∂ x u (0) , which is the longest gradient chain that we can obtain during back-propagation, stays constant over time.

Original: The backward propagation is then stable and non-dissipative, and no gradient vanishing or explosion can occur during training.
Revised: The backward propagation is then stable and non-dissipative, and no gradient vanishing or explosion can occur during training.

Original: C P ROOF OF P ROPOSITION
Revised: C P ROOF OF P ROPOSITION

Original: The Jacobian matrix of Equation 4 is given by:
Revised: The Jacobian matrix of Equation 4 is given by:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Following (Chang et al., 2019; 2018), we can see the right-hand side of Equation 11 as the result of a matrix multiplication between an invertible diagonal matrix and an anti-symmetric matrix.
Revised: Following (Chang et al., 2019; 2018), we can see the right-hand side of Equation 11 as the result of a matrix multiplication between an invertible diagonal matrix and an anti-symmetric matrix.

Original: Specifically, defining A = diag (cid:2) σ ′ (cid:0) ( W − W T ) x u ( t ) + Φ( X ( t ) , N u ) + b (cid:1)(cid:3) and B = W − W T , we have J ( t ) = AB .
Revised: Specifically, defining A = diag (cid:2) σ ′ (cid:0) ( W − W T ) x u ( t ) + Φ( X ( t ) , N u ) + b (cid:1)(cid:3) and B = W − W T , we have J ( t ) = AB .

Original: Let us now consider an eigenpair of AB , where the eigenvector is denoted by v and the eigenvalue by λ .
Revised: Let us now consider an eigenpair of AB , where the eigenvector is denoted by v and the eigenvalue by λ .

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: (12) where ∗ represents the conjugate transpose.
Revised: (12) where ∗ represents the conjugate transpose.

Original: On the right-hand side of Equation 12, we can notice that the ( v ∗ A − 1 v ) term is a real number.
Revised: On the right-hand side of Equation 12, we can notice that the ( v ∗ A − 1 v ) term is a real number.

Original: Recalling that B ∗ =
Revised: Recalling that B ∗ =

Original: B T = − B for a real anti-symmetric matrix, we can notice that ( v ∗ Bv ) ∗ =
Revised: B T = − B for a real anti-symmetric matrix, we can notice that ( v ∗ Bv ) ∗ =

Original: v ∗ B ∗ v = − v ∗ Bv .
Revised: v ∗ B ∗ v = − v ∗ Bv .

Original: Hence, the v ∗ Bv term on the left-hand side of Equation 12 is an imaginary number.
Revised: Hence, the v ∗ Bv term on the left-hand side of Equation 12 is an imaginary number.

Original: Thereby, λ needs to be purely imaginary, and, as a result, all eigenvalues of J ( t ) are purely imaginary.
Revised: Thereby, λ needs to be purely imaginary, and, as a result, all eigenvalues of J ( t ) are purely imaginary.

Original: D D ATASETS DESCRIPTION AND STATISTICS
Revised: D D ATASETS DESCRIPTION AND STATISTICS

Original: In the graph property prediction (GPP) experiments, we employed the same generation procedure as in Corso et al.
Revised: In the graph property prediction (GPP) experiments, we employed the same generation procedure as in Corso et al.

Original: Graphs are randomly sampled from several graph distributions, such as
Revised: Graphs are randomly sampled from several graph distributions, such as

Original: PubMed is a citation network where each node represents a paper and each edge indicates that one paper cites another one.
Revised: PubMed is a citation network where each node represents a paper and each edge indicates that one paper cites another one.

Original: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.
Revised: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.

Original: The class labels represent the papers categories.
Revised: The class labels represent the papers categories.

Original: Amazon Computers and Amazon Photo are portions of the Amazon co-purchase graph, where nodes represent goods and edges indicate that two goods are frequently bought together.
Revised: Amazon Computers and Amazon Photo are portions of the Amazon co-purchase graph, where nodes represent goods and edges indicate that two goods are frequently bought together.

Original: Node features are bag-of-words encoded product reviews, and class labels are given by the product category.
Revised: Node features are bag-of-words encoded product reviews, and class labels are given by the product category.

Original: Coauthor CS and Coauthor Physics are co-authorship graphs extracted from the Microsoft Academic
Revised: Coauthor CS and Coauthor Physics are co-authorship graphs extracted from the Microsoft Academic

Original: Graph 4 where nodes are authors, that are connected by an edge if they co-authored a paper.
Revised: Graph 4 where nodes are authors, that are connected by an edge if they co-authored a paper.

Original: Node features represent paper keywords for each author’s papers, and class labels indicate most active fields of study for each author.
Revised: Node features represent paper keywords for each author’s papers, and class labels indicate most active fields of study for each author.

Original: Table 3 contains the statistics of the employed datasets, sorted by graph density.
Revised: Table 3 contains the statistics of the employed datasets, sorted by graph density.

Original: The density of a graph is computed as the ratio between the number of edges and the number of possible edges, i.e., d = |E| |V| ( |V|− 1) .
Revised: The density of a graph is computed as the ratio between the number of edges and the number of possible edges, i.e., d = |E| |V| ( |V|− 1) .

Original: [Table]
Revised: [Table]

Original: E E XPLORED HYPER - PARAMETER SPACE
Revised: E E XPLORED HYPER - PARAMETER SPACE

Original: In Table 4 and Table 5 we report the grids of hyper-parameters employed in our experiments by each method.
Revised: In Table 4 and Table 5 we report the grids of hyper-parameters employed in our experiments by each method.

Original: We recall that the hyper-parameters ϵ and γ refer only to our method
Revised: We recall that the hyper-parameters ϵ and γ refer only to our method

Original: [Table] [Table]
Revised: [Table] [Table]

Original: F C OMPLETE RESULTS
Revised: F C OMPLETE RESULTS

Original: In Table 6 we report the complete results for the graph property prediction benchmark, including all the versions and cofiguration of our A-DGN.
Revised: In Table 6 we report the complete results for the graph property prediction benchmark, including all the versions and cofiguration of our A-DGN.

Original: We reported the average time per epoch (measured
Revised: We reported the average time per epoch (measured

Original: [Table]
Revised: [Table]

Original: in seconds) for each model in the graph property prediction task in Table 7.
Revised: in seconds) for each model in the graph property prediction task in Table 7.

Original: Each model in the evaluation has 20 layers and an embedding dimesion equal to 30.
Revised: Each model in the evaluation has 20 layers and an embedding dimesion equal to 30.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: G A DDITIONAL EXPERIMENTS G. G RAPH HETEROPHILIC BENCHMARKS
Revised: G A DDITIONAL EXPERIMENTS G. G RAPH HETEROPHILIC BENCHMARKS

Original: In the graph heterophilic benchmark we consider six well-known graph datasets for node classification, i [Table] e., Chameleon, Squirrel, Actor, Cornell, Texas, and Wisconsin.
Revised: In the graph heterophilic benchmark we consider six well-known graph datasets for node classification, i.e., Chameleon, Squirrel, Actor, Cornell, Texas, and Wisconsin.

Original: We employed the same experimental setting as Pei et al.
Revised: We employed the same experimental setting as Pei et al.

Original: As in the graph benchmark in Section 4.2, we study only the version of A-DGN with weight sharing.
Revised: As in the graph benchmark in Section 4.2, we study only the version of A-DGN with weight sharing.

Original: We perform hyper-parameter tuning via grid search, optimizing the accuracy score.
Revised: We perform hyper-parameter tuning via grid search, optimizing the accuracy score.

Original: We report in Table 8 the grid of hyper-parameters explored for this experiment.
Revised: We report in Table 8 the grid of hyper-parameters explored for this experiment.

Original: [Table]
Revised: [Table]

Original: We present the results on the graph heterophilic benchmark in Table 9.
Revised: We present the results on the graph heterophilic benchmark in Table 9.

Original: Specifically, we report the accuracy as the evaluation metric.
Revised: Specifically, we report the accuracy as the evaluation metric.

Original: Even in this scenario, A-DGN outperforms the selected baselines on all tasks except one (Chameleon).
Revised: Even in this scenario, A-DGN outperforms the selected baselines on all tasks except one (Chameleon).

Original: Indeed, A-DGN shows a decisive improvement of, on average, 3.6% with respect to the best baseline in each task.
Revised: Indeed, A-DGN shows a decisive improvement of, on average, 3.6% with respect to the best baseline in each task.

Original: The maximum is reached in the Wisconsin dataset, where the improvement is 5.6%.
Revised: The maximum is reached in the Wisconsin dataset, where the improvement is 5.6%.

Original: Similarly to the graph benchmarks in Section 4.2, our approach maintains or improves the performance as the number of layers increases, as Figure 3 shows.
Revised: Similarly to the graph benchmarks in Section 4.2, our approach maintains or improves the performance as the number of layers increases, as Figure 3 shows.

Original: Moreover, in this experiment, we show that A-DGN has outstanding performance even with 64 layers.
Revised: Moreover, in this experiment, we show that A-DGN has outstanding performance even with 64 layers.

Original: Thus, A-DGN is able to effectively propagate the long-range information between nodes even in the scenario of graphs with high heterophilic levels.
Revised: Thus, A-DGN is able to effectively propagate the long-range information between nodes even in the scenario of graphs with high heterophilic levels.

Original: Such result suggests that the presented approach can be a starting point to mitigate the over-smoothing problem as well.
Revised: Such result suggests that the presented approach can be a starting point to mitigate the over-smoothing problem as well.

Original: The higher the better.

Original: The ” ⋄ ” results are obtained from Pei et al.

Original: (2020), the ” △ ” results are obtained from Ming Chen et al.

Original: (2020), and “N/A” denotes non-reported results.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: G. T REE -N EIGHBORS M ATCH

Original: [Table] [Table] [Table] [Table]
Revised: In the Tree-NeighborsMatch task we consider the same experimental setting as Alon & Yahav (2021).

Original: [Table] [Table] [Table] [Table]
Revised: For simplicity, we focus on the version of A-DGN with weight sharing and simple aggregation scheme (Equation 6).

Original: [Table] [Table] [Table] [Table]
Revised: We report in Table 10 the grid of hyper-parameters explored for this experiment.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: We present the results on the Tree-NeighborsMatch benchmark in Table 11.

Original: [Table] [Table] [Table] [Table]
Revised: In this case, to avoid having an overly unbalanced setup in terms of the number of trainable parameters, we have increased the number of hidden unit in our method (we use weight sharing, while baselines present up totrainable layers).

Original: [Table] [Table] [Table] [Table]
Revised: Moreover, we have normalized the achieved accuracy, reporting the ratio between the accuracy score in percentage points and the employed total number of trainable hidden units, i.e., acc × 100 /N tot .

Original: [Table] [Table] [Table] [Table]
Revised: As it can be seen, A-DGN generally outperforms the baselines (with the only exception of radius = 5).

Original: [Table] [Table] [Table] [Table]
Revised: Moreover, even considering the un-normalized (i.e., original) accuracy, we observe that, even with fewer trainable parameters, A-DGN achieves on par or better performance compared to GAT, GCN, and GIN.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]