Original: L ATENT G RAPH I NFERENCE USING P RODUCT M ANIFOLDS
Revised: L ATENT G RAPH I NFERENCE USING P RODUCT M ANIFOLDS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph Neural Networks usually rely on the assumption that the graph topology is available to the network as well as optimal for the downstream task.
Revised: Graph Neural Networks usually rely on the assumption that the graph topology is available to the network as well as optimal for the downstream task.

Original: Latent graph inference allows models to dynamically learn the intrinsic graph structure of problems where the connectivity patterns of data may not be directly accessible.
Revised: Latent graph inference allows models to dynamically learn the intrinsic graph structure of problems where the connectivity patterns of data may not be directly accessible.

Original: In this work, we generalize the discrete Differentiable Graph Module (dDGM) for latent graph learning.
Revised: In this work, we generalize the discrete Differentiable Graph Module (dDGM) for latent graph learning.

Original: The original dDGM architecture used the Euclidean plane to encode latent features based on which the latent graphs were generated.
Revised: The original dDGM architecture used the Euclidean plane to encode latent features based on which the latent graphs were generated.

Original: By incorporating Riemannian geometry into the model and generating more complex embedding spaces, we can improve the performance of the latent graph inference system.
Revised: By incorporating Riemannian geometry into the model and generating more complex embedding spaces, we can improve the performance of the latent graph inference system.

Original: In particular, we propose a computationally tractable approach to produce product manifolds of constant curvature model spaces that can encode latent features of varying structure.
Revised: In particular, we propose a computationally tractable approach to produce product manifolds of constant curvature model spaces that can encode latent features of varying structure.

Original: The latent representations mapped onto the inferred product manifold are used to compute richer similarity measures that are leveraged by the latent graph learning model to obtain optimized latent graphs.
Revised: The latent representations mapped onto the inferred product manifold are used to compute richer similarity measures that are leveraged by the latent graph learning model to obtain optimized latent graphs.

Original: Moreover, the curvature of the product manifold is learned during training alongside the rest of the network parameters and based on the downstream task, rather than it being a static embedding space.
Revised: Moreover, the curvature of the product manifold is learned during training alongside the rest of the network parameters and based on the downstream task, rather than it being a static embedding space.

Original: Our novel approach is tested on a wide range of datasets, and outperforms the original dDGM model.
Revised: Our novel approach is tested on a wide range of datasets, and outperforms the original dDGM model.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph Neural Networks (GNNs) have achieved state-of-the-art performance in a number of applications, from travel-time prediction (Derrow-Pinion et al.
Revised: Graph Neural Networks (GNNs) have achieved state-of-the-art performance in a number of applications, from travel-time prediction (Derrow-Pinion et al.

Original: (2021)) to antibiotic discovery (Stokes et al.
Revised: (2021)) to antibiotic discovery (Stokes et al.

Original: They leverage the connectivity structure of graph data, which improves their performance in many applications as compared to traditional neural networks (Bronstein et al.
Revised: They leverage the connectivity structure of graph data, which improves their performance in many applications as compared to traditional neural networks (Bronstein et al.

Original: Most current GNN architectures assume that the topology of the graph is given and fixed during training.
Revised: Most current GNN architectures assume that the topology of the graph is given and fixed during training.

Original: Hence, they update the input node features, and sometimes edge features, but preserve the input graph topology.
Revised: Hence, they update the input node features, and sometimes edge features, but preserve the input graph topology.

Original: A substantial amount of research has focused on improving diffusion using different types of GNN layers.
Revised: A substantial amount of research has focused on improving diffusion using different types of GNN layers.

Original: However, discovering an optimal graph topology that can help diffusion has only recently gained attention (Topping et al.
Revised: However, discovering an optimal graph topology that can help diffusion has only recently gained attention (Topping et al.

Original: ; Cosmo et al.
Revised: ; Cosmo et al.

Original: Kazi et al.
Revised: Kazi et al.

Original: In many real-world applications, data can have some underlying but unknown graph structure, which we call a latent graph .
Revised: In many real-world applications, data can have some underlying but unknown graph structure, which we call a latent graph .

Original: That is, we may only be able to access a pointcloud of data.
Revised: That is, we may only be able to access a pointcloud of data.

Original: Nevertheless, this does not necessarily mean the data is not intrinsically related, and that its connectivity cannot be leveraged to make more accurate predictions.
Revised: Nevertheless, this does not necessarily mean the data is not intrinsically related, and that its connectivity cannot be leveraged to make more accurate predictions.

Original: The vast majority of Geometric Deep Learning research so far has relied on human annotators or simplistic pre-processing algorithms to generate the graph structure to be passed to GNNs.
Revised: The vast majority of Geometric Deep Learning research so far has relied on human annotators or simplistic pre-processing algorithms to generate the graph structure to be passed to GNNs.

Original: Furthermore, in practice, even in settings where the correct graph is provided, it may often be suboptimal for the task at hand, and the GNN may benefit from rewiring (Topping et al.
Revised: Furthermore, in practice, even in settings where the correct graph is provided, it may often be suboptimal for the task at hand, and the GNN may benefit from rewiring (Topping et al.

Original: In this work, we drop the assumption that the graph adjacency matrix is given and study how to learn the latent graph in a fully-differentiable manner, using product manifolds, alongside the GNN diffusion layers.
Revised: In this work, we drop the assumption that the graph adjacency matrix is given and study how to learn the latent graph in a fully-differentiable manner, using product manifolds, alongside the GNN diffusion layers.

Original: More elaborately, we incorporate Riemannian geometry to the discrete Differentiable Graph Module (dDGM) proposed by Kazi et al.
Revised: More elaborately, we incorporate Riemannian geometry to the discrete Differentiable Graph Module (dDGM) proposed by Kazi et al.

Original: We show that it is possible and beneficial to encode latent features into more complex embedding spaces beyond the Euclidean plane used in the original work.
Revised: We show that it is possible and beneficial to encode latent features into more complex embedding spaces beyond the Euclidean plane used in the original work.

Original: In particular, we leverage the convenient mathematical properties of product manifolds to learn the curvature of the embedding space in a fully-differentiable manner.
Revised: In particular, we leverage the convenient mathematical properties of product manifolds to learn the curvature of the embedding space in a fully-differentiable manner.

Original: Contributions 1)
Revised: Contributions 1)

Original: We explain how to use model spaces of constant curvature for the embedding space.
Revised: We explain how to use model spaces of constant curvature for the embedding space.

Original: To do so, we outline a principled procedure to map Euclidean GNN output features to constant curvature model space manifolds with non-zero curvature: we use the hypersphere for spherical space and the hyperboloid for hyperbolic space.
Revised: To do so, we outline a principled procedure to map Euclidean GNN output features to constant curvature model space manifolds with non-zero curvature: we use the hypersphere for spherical space and the hyperboloid for hyperbolic space.

Original: We also outline how to calculate distances between points in these spaces, which are then used by the dDGM sparse graph generating procedure to infer the edges of the latent graph.
Revised: We also outline how to calculate distances between points in these spaces, which are then used by the dDGM sparse graph generating procedure to infer the edges of the latent graph.

Original: Unlike the original dDGM model which explored using the
Revised: Unlike the original dDGM model which explored using the

Original: Poincar´e ball with fixed curvature for modeling hyperbolic space, in this work we use hyperboloids of arbitrary negative curvature.
Revised: Poincar´e ball with fixed curvature for modeling hyperbolic space, in this work we use hyperboloids of arbitrary negative curvature.

Original: We show how to construct more complex embedding spaces that can encode latent data of varying structure using product manifolds of model spaces.
Revised: We show how to construct more complex embedding spaces that can encode latent data of varying structure using product manifolds of model spaces.

Original: The curvature of each model space composing the product manifold is learned in a fully-differentiable manner alongside the rest of the model parameters, and based on the downstream task performance.
Revised: The curvature of each model space composing the product manifold is learned in a fully-differentiable manner alongside the rest of the model parameters, and based on the downstream task performance.

Original: We test our approach on 15 datasets which includes standard homophilic graph datasets, heterophilic graphs, large-scale graphs, molecular datasets, and datasets for other real-world applications such as brain imaging and aerospace engineering.
Revised: We test our approach on 15 datasets which includes standard homophilic graph datasets, heterophilic graphs, large-scale graphs, molecular datasets, and datasets for other real-world applications such as brain imaging and aerospace engineering.

Original: 4) It has been shown that traditional GNN models, such as Graph Convolutional Networks (GCNs) (Kipf & Welling (2017)) and Graph Attention Networks (GATs) (Veliˇckovi´c et al.
Revised: 4) It has been shown that traditional GNN models, such as Graph Convolutional Networks (GCNs) (Kipf & Welling (2017)) and Graph Attention Networks (GATs) (Veliˇckovi´c et al.

Original: (2018)) struggle to achieve good performance in heterophilic datasets (Zhu et al.
Revised: (2018)) struggle to achieve good performance in heterophilic datasets (Zhu et al.

Original: (2020)), since in fact homophily is used as an inductive bias by these models.
Revised: (2020)), since in fact homophily is used as an inductive bias by these models.

Original: Amongst other models, Sheaf Neural Networks (SNNs) (Hansen & Gebhart (2020); Bodnar et al. (2022);
Revised: Amongst other models, Sheaf Neural Networks (SNNs) (Hansen & Gebhart (2020); Bodnar et al. (2022);

Original: Barbero et al.
Revised: Barbero et al.

Original: (2022)) have been proposed to tackle this issue.
Revised: (2022b;a)) have been proposed to tackle this issue.

Original: We show that latent graph inference enables traditional GNN models to give good performance on heterophilic datasets without having to resort to sophisticated diffusion layers or model architectures such as SNNs. 5)
Revised: We show that latent graph inference enables traditional GNN models to give good performance on heterophilic datasets without having to resort to sophisticated diffusion layers or model architectures such as SNNs. 5)

Original: To make this work accessible to the wider machine learning community, we have created a new PyTorch Geometric layer.
Revised: To make this work accessible to the wider machine learning community, we have created a new PyTorch Geometric layer.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section we discuss relevant background for this work.
Revised: In this section we discuss relevant background for this work.

Original: We first provide a literature review regarding recent advances in latent graph inference using GNNs as well as related work on manifold learning and graph embedding.
Revised: We first provide a literature review regarding recent advances in latent graph inference using GNNs as well as related work on manifold learning and graph embedding.

Original: Next, we give an overview of the original Differentiable Graph
Revised: Next, we give an overview of the original Differentiable Graph

Original: Module (DGM) formulation, but we recommend referring to Kazi et al.
Revised: Module (DGM) formulation, but we recommend referring to Kazi et al.

Original: for further details.
Revised: for further details.

Original: 2. R ELATED W ORK
Revised: 2. R ELATED W ORK

Original: Latent graph and topology inference is a standing problem in Geometric Deep Learning.
Revised: Latent graph and topology inference is a standing problem in Geometric Deep Learning.

Original: In contrast to algorithms that work on sets and that apply a shared pointwise function such as PointNet (Qi et al.
Revised: In contrast to algorithms that work on sets and that apply a shared pointwise function such as PointNet (Qi et al.

Original: (2017)), in latent graph inference we want to learn to optimally share information between nodes in the pointcloud.
Revised: (2017)), in latent graph inference we want to learn to optimally share information between nodes in the pointcloud.

Original: Some contributions in the literature have focused on applying pre-processing steps to enhance diffusion based on an initial input graph (Topping et al.
Revised: Some contributions in the literature have focused on applying pre-processing steps to enhance diffusion based on an initial input graph (Topping et al.

Original: ; Gasteiger et al.
Revised: ; Gasteiger et al.

Original:  Alon & Yahav (2021)).
Revised: (2019); Alon & Yahav (2021) ; Wu et al.

Original: Note, however, that this area of research focuses on improving an already existing graph which may be suboptimal for the downstream task.
Revised: Note, however, that this area of research focuses on improving an already existing graph which may be suboptimal for the downstream task.

Original: This paper is more directly related to work that addresses how to learn the graph topology dynamically, instead of assuming a fixed graph at the start of training.
Revised: This paper is more directly related to work that addresses how to learn the graph topology dynamically, instead of assuming a fixed graph at the start of training.

Original: When the underlying connectivity structure is unknown, architectures such as transformers (Vaswani et al.
Revised: When the underlying connectivity structure is unknown, architectures such as transformers (Vaswani et al.

Original: and attentional multi-agent predictive models (Hoshen (2017)), simply assume the graph to be fully-connected, but this can become hard to scale to large graphs.
Revised: and attentional multi-agent predictive models (Hoshen (2017)), simply assume the graph to be fully-connected, but this can become hard to scale to large graphs.

Original: Generating sparse graphs can result in more computationally tractable solutions (Fetaya et al.
Revised: Generating sparse graphs can result in more computationally tractable solutions (Fetaya et al.

Original: and avoid over-smoothing (Chen et al.
Revised: and avoid over-smoothing (Chen et al.

Original: (2020a)).
Revised: (2020a)).

Original: For this a series of models have been proposed, starting from Dynamic Graph Convolutional Neural Networks (DGCNNs)
Revised: For this a series of models have been proposed, starting from Dynamic Graph Convolutional Neural Networks (DGCNNs)

Original: (Wang et al.
Revised: (Wang et al.

Original: (2019)), to other solutions that decouple graph inference and information diffusion, such as the Differentiable Graph Modules (DGMs) in Cosmo et al.
Revised: (2019)), to other solutions that decouple graph inference and information diffusion, such as the Differentiable Graph Modules (DGMs) in Cosmo et al.

Original: (2020) and Kazi et al.
Revised: (2020) and Kazi et al.

Original: Note that latent graph inference may also be referred to as graph structure learning in the literature.
Revised: Note that latent graph inference may also be referred to as graph structure learning in the literature.

Original: A survey of similar methods can be found in Zhu et al.
Revised: A survey of similar methods can be found in Zhu et al.

Original: (2021), and some additional classical methods include LDS-GNN (Franceschi et al., 2019), IDGL (Chen et al., 2020b), and Pro-GNN (Jin et al., 2020).
Revised: (2021), and some additional classical methods include LDS-GNN (Franceschi et al., 2019), IDGL (Chen et al., 2020b), and ProGNN (Jin et al., 2020).

Original: In this work, we extend the dDGM module proposed by Kazi et al.
Revised: In this work, we extend the dDGM module proposed by Kazi et al.

Original: for learning latent graphs using product manifolds.
Revised: for learning latent graphs using product manifolds.

Original: Product spaces have primarily been studied in the manifold learning and graph embedding literature (Cayton (2005); Fefferman et al.
Revised: Product spaces have primarily been studied in the manifold learning and graph embedding literature (Cayton (2005); Fefferman et al.

Original: ; Bengio et al.
Revised: ; Bengio et al.

Original: Recent work has started exploring encoding the geometry of data into rich ambient manifolds.
Revised: Recent work has started exploring encoding the geometry of data into rich ambient manifolds.

Original: In particular, hyperbolic geometry has proven successful in a number of tasks (Liu et al.
Revised: In particular, hyperbolic geometry has proven successful in a number of tasks (Liu et al.

Original: ; Chamberlain et al.
Revised: ; Chamberlain et al.

Original: (2017); Sala et al.
Revised: (2017); Sala et al.

Original: Different manifold classes have been employed to enhance modeling flexibility, such as products of constant curvature spaces (Gu et al.
Revised: Different manifold classes have been employed to enhance modeling flexibility, such as products of constant curvature spaces (Gu et al.

Original: , matrix manifolds (Cruceru et al.
Revised: , matrix manifolds (Cruceru et al.

Original: (2021)), and heterogeneous manifolds (Di Giovanni et al.
Revised: (2021)), and heterogeneous manifolds (Di Giovanni et al.

Original: We will leverage these ideas and use product manifolds to generate the embedding space for constructing our latent graphs.
Revised: We will leverage these ideas and use product manifolds to generate the embedding space for constructing our latent graphs.

Original:  A N O VERVIEW OF THE D ISCRETE D IFFERENTIABLE G RAPH M ODULE
Revised: 2. A N O VERVIEW OF THE D ISCRETE D IFFERENTIABLE G RAPH M ODULE

Original: Kazi et al.
Revised: Kazi et al.

Original: (2022) proposed a general technique for learning an optimized latent graph, based on the output features of each layer, onto which to apply the downstream GNN diffusion layers.
Revised: (2022) proposed a general technique for learning an optimized latent graph, based on the output features of each layer, onto which to apply the downstream GNN diffusion layers.

Original: Here, we specifically focus on the dDGM module (not the cDGM), which is much more computationally efficient and recommended by the authors.
Revised: Here, we specifically focus on the dDGM module (not the cDGM), which is much more computationally efficient and recommended by the authors.

Original: The main idea is to use some measure of similarity between the latent node features to generate latent graphs which are optimal for each layer l .
Revised: The main idea is to use some measure of similarity between the latent node features to generate latent graphs which are optimal for each layer l .

Original: We can summarize the architecture as ˆX p l ` 1 q “ f p l q Θ p concat p
Revised: We can summarize the architecture as ˆX p l ` 1 q “ f p l q Θ p concat p

Original: X p l q , ˆX 
Revised: X p l q , ˆX p

Original: [Equation] l q [Equation]
Revised: l q q , A p l q q Ñ A p l ` 1 q „ P p l q p ˆX p l ` 1 q qq Ñ

Original: X p l ` 1 q “ g ϕ p X p l q , A p l ` 1 q q .
Revised: X p l ` 1 q “ g ϕ p X p l q , A p l ` 1 q q .

Original: The node features in layer l , X p l q , are transformed into ˆX p l ` 1 q through a function f
Revised: The node features in layer l , X p l q , are transformed into ˆX p l ` 1 q through a function f

Original: Θ p l q , which has learnable parameters, and compared using a similarity measure φ p T q , which is parameterized by a scalar learnable parameter T .
Revised: Θ p l q , which has learnable parameters, and compared using a similarity measure φ p T q , which is parameterized by a scalar learnable parameter T .

Original: On the other hand, g ϕ is a diffusion function which in practice corresponds to multiple GNN layers stacked together.
Revised: On the other hand, g ϕ is a diffusion function which in practice corresponds to multiple GNN layers stacked together.

Original: g ϕ diffuses information based on the inferred latent graph connectivity structure summarized in A p l ` 1 q , which is an unweighted sparse matrix.
Revised: g ϕ diffuses information based on the inferred latent graph connectivity structure summarized in A p l ` 1 q , which is an unweighted sparse matrix.

Original: The dDGM module generates a sparse k -degree graph using the Gumbel
Revised: The dDGM module generates a sparse k -degree graph using the Gumbel

Original: Top-k trick (Kool et al.
Revised: Top-k trick (Kool et al.

Original: (2019)), a stochastic relaxation of the kNN rule, to sample edges from the probability matrix P
Revised: (2019)), a stochastic relaxation of the kNN rule, to sample edges from the probability matrix P

Original: p l q p X p l q ; Θ p l q , T q , where each entry corresponds to
Revised: p l q p X p l q ; Θ p l q , T q , where each entry corresponds to

Original: [Equation]
Revised: [Equation]

Original: The main similarity measure used in Kazi et al.
Revised: The main similarity measure used in Kazi et al.

Original: (2022) was to compute the distance based on the features of two nodes in the graph embedding space.
Revised: (2022) was to compute the distance based on the features of two nodes in the graph embedding space.

Original: They assumed that the latent features laid in an
Revised: They assumed that the latent features laid in an

Original: Euclidean plane of constant curvature K E “ 0 , so that p p
Revised: Euclidean plane of constant curvature K E “ 0 , so that p p

Original: l q ij “ exp p´T d E p
Revised: l q ij “ exp p´

Original: f
Revised: [Equation] “ exp [Equation]

Original: p
Revised: d

Original: l q Θ p x p l q i q , f p l q Θ p x p l q j qqq “ exp
Revised: [Equation] p ˆx p l `

Original: [Equation]
Revised: [Equation]

Original:  , ˆx p 
Revised: i , ˆx p l `

Original: [Equation]
Revised: [Equation]

Original:  qq , where d E denotes distance in Euclidean space.
Revised: j qq , where d E denotes distance in Euclidean space.

Original: Then, based on argsort p log p p
Revised: Then, based on argsort p log p p

Original: p l q i q ´ log p´ log p q qqq , where q P R N is uniform i.i.d in the interval r 0 , 1 s , we can sample the edges E p l q p X p l q ; Θ p l q , T, k q “ tp i, j i, 1 q , p i, j i, 2 q , ..., p i, j i,k q : i “ 1 , ..., N u , where k is the number of sampled connections using the Gumbel Top-k trick.
Revised: p l q i q ´ log p´ log p q qqq , where q P R N is uniform i.i.d in the interval r 0 , 1 s , we can sample the edges E p l q p X p l q ; Θ p l q , T, k q “ tp i, j i, 1 q , p i, j i, 2 q , ..., p i, j i,k q : i “ 1 , ..., N u , where k is the number of sampled connections using the Gumbel Top-k trick.

Original: This sampling approach follows the categorical distribution p p l q ij Σ r p p l q ir and E p X p l q ; Θ p l q , T, k q is represented by the unweighted adjacency matrix A p l q p X p l q ; Θ p l q , T, k q .
Revised: This sampling approach follows the categorical distribution p p l q ij Σ r p p l q ir and E p X p l q ; Θ p l q , T, k q is represented by the unweighted adjacency matrix A p l q p X p l q ; Θ p l q , T, k q .

Original: Note that including noise in the edge sampling approach will result in the generation of some random edges in the latent graphs which can be understood as a form of regularization.
Revised: Note that including noise in the edge sampling approach will result in the generation of some random edges in the latent graphs which can be understood as a form of regularization.

Original: In this work, we generalize Equation 1 to measure similarities based on distances but dropping the assumption used in Kazi et al.
Revised: In this work, we generalize Equation 1 to measure similarities based on distances but dropping the assumption used in Kazi et al.

Original: (2022), in which they limit themselves to fixed-curvature spaces, specifically to Eucliden space where K E “ 0 .
Revised: (2022), in which they limit themselves to fixed-curvature spaces, specifically to Eucliden space where K E “ 0 .

Original: We will use product manifolds of model spaces of constant curvature to improve the similarity measure φ and construct better latent graphs.
Revised: We will use product manifolds of model spaces of constant curvature to improve the similarity measure φ and construct better latent graphs.

Original:  M ETHOD : P RODUCT M ANIFOLDS FOR L ATENT G RAPH I NFERENCE
Revised:  M ETHOD : P RODUCT M ANIFOLDS FOR L ATENT G RAPH I NFERENCE

Original: In this section, we first introduce model spaces, which are a special type of Riemannian manifolds, and explain how to map Euclidean GNN output features to model spaces with non-zero curvature.
Revised: In this section, we first introduce model spaces, which are a special type of Riemannian manifolds, and explain how to map Euclidean GNN output features to model spaces with non-zero curvature.

Original: In case the reader is unfamiliar with the topic, additional details regarding Riemannian manifolds can be found in Appendix A.
Revised: In case the reader is unfamiliar with the topic, additional details regarding Riemannian manifolds can be found in Appendix A.

Original: Then, we mathematically define product manifolds and how to calculate distances between points in the manifold.
Revised: Then, we mathematically define product manifolds and how to calculate distances between points in the manifold.

Original: Next, we introduce scaling metrics which help us learn the curvature of each model space composing the product manifold.
Revised: Next, we introduce scaling metrics which help us learn the curvature of each model space composing the product manifold.

Original: A discussion on product manifold curvature learning can be found in Appendix B.
Revised: A discussion on product manifold curvature learning can be found in Appendix B.

Original: The intuition behind the method is that we can consider the embedding space represented by the product manifold as a combination of more simple spaces (model spaces of constant curvature), and compute distances between the latent representations mapped onto the product manifold by considering distances in each model space individually and later aggregating them in a principled manner.
Revised: The intuition behind the method is that we can consider the embedding space represented by the product manifold as a combination of more simple spaces (model spaces of constant curvature), and compute distances between the latent representations mapped onto the product manifold by considering distances in each model space individually and later aggregating them in a principled manner.

Original: This allows to generate diverse embedding spaces which at the same time are computationally tractable.
Revised: This allows to generate diverse embedding spaces which at the same time are computationally tractable.

Original: 3. C ONSTANT C URVATURE M ODEL S PACES
Revised: 3. C ONSTANT C URVATURE M ODEL S PACES

Original: Curvature is effectively a measure of geodesic dispersion.
Revised: Curvature is effectively a measure of geodesic dispersion.

Original: When there is no curvature geodesics stay parallel, with negative curvature they diverge, and with positive curvature they converge.
Revised: When there is no curvature geodesics stay parallel, with negative curvature they diverge, and with positive curvature they converge.

Original: Euclidean space, E d E K E “ R d E , is a flat space with curvature K E “ 0 .
Revised: Euclidean space, E d E K E “ R d E , is a flat space with curvature K E “ 0 .

Original: Note that here we use d E to denote dimensionality.
Revised: Note that here we use d E to denote dimensionality.

Original: On the other hand, hyperbolic and spherical space, have negative and positive curvature, respectively.
Revised: On the other hand, hyperbolic and spherical space, have negative and positive curvature, respectively.

Original: We define hyperboloids as H d H K H “ t x p P R d H ` 1 : x x p , x p y L “ 1 { K H u , where
Revised: We define hyperboloids as H d H K H “ t x p P R d H ` 1 : x x p , x p y L “ 1 { K H u , where

Original: K H ă 0 and x¨ , ¨y L is the Lorentz inner product x
Revised: K H ă 0 and x¨ , ¨y L is the Lorentz inner product x

Original: x , y y L “ ´ x 1 y 1 `
Revised: x , y y L “ ´ x 1 y 1 `

Original: [Equation]
Revised: [Equation]

Original: and hyperspheres as S d S K S “ t x p P R d S ` 1 : x x p , x p y 2 “ 1 { K S u , where K S ą 0 and x¨ , ¨y 2 is the standard Euclidean inner product x x , y y 2 “ ř d S ` 1 j “ 1 x j y j , @ x , y P R d S ` 1 .
Revised: and hyperspheres as S d S K S “ t x p P R d S ` 1 : x x p , x p y 2 “ 1 { K S u , where K S ą 0 and x¨ , ¨y 2 is the standard Euclidean inner product x x , y y 2 “ ř d S ` 1 j “ 1 x j y j , @ x , y P R d S ` 1 .

Original: Table 1 provides a summary of relevant operators in Euclidean, hyperbolic, and spherical spaces with arbitrary curvatures.
Revised: Table 1 provides a summary of relevant operators in Euclidean, hyperbolic, and spherical spaces with arbitrary curvatures.

Original: The closed forms for the distances between points in hyperbolic and spherical space use the arccosh and the arccos, their domains are t x P R : x ě 1 u and t x P R : ´ 1 ď x ď 1 u , respectively.
Revised: The closed forms for the distances between points in hyperbolic and spherical space use the arccosh and the arccos, their domains are t x P R : x ě 1 u and t x P R : ´ 1 ď x ď 1 u , respectively.

Original: We apply clipping to avoid giving inputs close to the domain limits and prevent from instabilities during training.
Revised: We apply clipping to avoid giving inputs close to the domain limits and prevent from instabilities during training.

Original: [Table]
Revised: [Table]

Original: | x || ˘
Revised: | x || ˘

Original: [Equation]
Revised: [Equation]

Original: hypersphere
Revised: hypersphere

Original: [Equation]
Revised: [Equation]

Original: The latent output features produced by the neural network layers are in Euclidean space and must be mapped to the relevant model spaces before applying the distance metrics.
Revised: The latent output features produced by the neural network layers are in Euclidean space and must be mapped to the relevant model spaces before applying the distance metrics.

Original: We use the appropriate exponential map (refer to Table 1).
Revised: We use the appropriate exponential map (refer to Table 1).

Original: To map Euclidean data to the hyperboloid, we use the hyperboloid north pole, that is, the origin o H K H : “ p 1 ?
Revised: To map Euclidean data to the hyperboloid, we use the hyperboloid north pole, that is, the origin o H K H : “ p 1 ?

Original: ´ K H , 0 , ..., 0 q “ p 1 ?
Revised: ´ K H , 0 , ..., 0 q “ p 1 ?

Original: ´
Revised: ´

Original: K H , 0 q as a reference point to perform tangent space operations.
Revised: K H , 0 q as a reference point to perform tangent space operations.

Original: Using the trick described in Chami et al.
Revised: Using the trick described in Chami et al.

Original: (2019) 1 , if x is an Euclidean feature we can consider concat p 0 , x q to be a point in the manifold tangent space at o H K H .
Revised: (2019) 1 , if x is an Euclidean feature we can consider concat p 0 , x q to be a point in the manifold tangent space at o H K H .

Original: Therefore, we can obtain the mapped features x “ exp H o H K H p concat p 0 , x qq via x “ ´
Revised: Therefore, we can obtain the mapped features x “ exp H o H K H p concat p 0 , x qq via x “ ´

Original: [Equation]
Revised: [Equation]

Original: Similarly for the hypersphere using as reference point
Revised: Similarly for the hypersphere using as reference point

Original: [Equation]
Revised: [Equation]

Original: 3. P RODUCT M ANIFOLDS
Revised: 3. P RODUCT M ANIFOLDS

Original: We define a product manifold as the Cartesian product P “ Ś n P i “ 1 M d i K i , where K i and d i are the curvature and dimensionality of the manifold M d
Revised: We define a product manifold as the Cartesian product P “ Ś n P i “ 1 M d i K i , where K i and d i are the curvature and dimensionality of the manifold M d

Original: i K i , respectively.
Revised: i K i , respectively.

Original: We write points x p P P using their coordinates x p “ concat
Revised: We write points x p P P using their coordinates x p “ concat

Original: [Equation]
Revised: [Equation]

Original: p 2 q p , ..., x p n P q p ¯ : x p i q p P M d
Revised: p 2 q p , ..., x p n P q p ¯ : x p i q p P M d

Original: i K i .
Revised: i K i .

Original: Also, the metric of the product manifold decomposes into the sum of the constituent metrics g P “ ř n P i “ 1 g i , hence, p P , g P q is also a Riemannian manifold if p M d i K i , g i q , @
Revised: Also, the metric of the product manifold decomposes into the sum of the constituent metrics g P “ ř n P i “ 1 g i , hence, p P , g P q is also a Riemannian manifold if p M d i K i , g i q , @

Original: i are all Riemannian manifolds in the first place.
Revised: i are all Riemannian manifolds in the first place.

Original: Note that the signature of the product space, that is, its parametrization, has several degrees of freedom: the number of components used, as well as the type of model spaces, their dimensionality, and curvature.
Revised: Note that the signature of the product space, that is, its parametrization, has several degrees of freedom: the number of components used, as well as the type of model spaces, their dimensionality, and curvature.

Original: If we restrict P to be composed of the Euclidean plane E d E K E , hyperboloids H d H j K H j , and hyperspheres
Revised: If we restrict P to be composed of the Euclidean plane E d E K E , hyperboloids H d H j K H j , and hyperspheres

Original: S d S k K S k of constant curvature, we can write an arbitrary product manifold of model spaces as
Revised: S d S k K S k of constant curvature, we can write an arbitrary product manifold of model spaces as

Original: [Equation]
Revised: [Equation]

Original: where K E “ 0 ,
Revised: where K E “ 0 ,

Original: K H
Revised: K H

Original: j ă 0 , and
Revised: j ă 0 , and

Original: K S k ą 0 .
Revised: K S k ą 0 .

Original: The rightmost part of Equation 2 is included to simplify the notation.
Revised: The rightmost part of Equation 2 is included to simplify the notation.

Original: P would have a total of 1 ` n H ` n S component spaces, and total dimension d E ` Σ n H j “ 1 d H j ` Σ n S k “ 1 d S k .
Revised: P would have a total of 1 ` n H ` n S component spaces, and total dimension d E ` Σ n H j “ 1 d H j ` Σ n S k “ 1 d S k .

Original: As shown in Gallier & Quaintance (2020), in the case of a product manifold as defined in Equation 2, the geodesics, exponential, and logarithmic maps on P are the concatenation of the corresponding notions of the individual model spaces.
Revised: As shown in Gallier & Quaintance (2020), in the case of a product manifold as defined in Equation 2, the geodesics, exponential, and logarithmic maps on P are the concatenation of the corresponding notions of the individual model spaces.

Original: 3. D ISTANCES AND S CALING M ETRICS FOR P RODUCT M ANIFOLDS
Revised: 3. D ISTANCES AND S CALING M ETRICS FOR P RODUCT M ANIFOLDS

Original: To compute distances between points in the product manifold we can add up the square distances for the coordinates in each of the individual manifolds
Revised: To compute distances between points in the product manifold we can add up the square distances for the coordinates in each of the individual manifolds

Original: d P p
Revised: d P p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ´
Revised: ´

Original: [Equation]
Revised: [Equation]

Original: ¯
Revised: ¯

Original: [Equation]
Revised: [Equation]

Original: , where the overline denotes that the adequate exponential map to project Euclidean feature entries to the relevant model space has been applied before computing the distance.
Revised: , where the overline denotes that the adequate exponential map to project Euclidean feature entries to the relevant model space has been applied before computing the distance.

Original: In practice, this would be equivalent to mapping the feature outputs to the product manifold and operating on P directly.
Revised: In practice, this would be equivalent to mapping the feature outputs to the product manifold and operating on P directly.

Original: As suggested in Tabaghi et al.
Revised: As suggested in Tabaghi et al.

Original: (2021), instead of directly updating the curvature of the hyperboloid and hypersphere model spaces used to construct the product manifold, we can set K H j “ ´ 1 , @
Revised: (2021), instead of directly updating the curvature of the hyperboloid and hypersphere model spaces used to construct the product manifold, we can set K H j “ ´ 1 , @

Original: j , and K S k “ 1 , @ k, and use a scaled distance metric instead.
Revised: j , and K S k “ 1 , @ k, and use a scaled distance metric instead.

Original: To do so, we introduce learnable coefficients α H j and α S k , d P p x p 1 , x p 2 q 2 “ d E
Revised: To do so, we introduce learnable coefficients α H j and α S k , d P p x p 1 , x p 2 q 2 “ d E

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to learning the curvature of the non-Euclidean model spaces, but computationally more tractable and efficient (for further details on how this coefficients are updated refer to Appendix C.2).
Revised: which is equivalent to learning the curvature of the non-Euclidean model spaces, but computationally more tractable and efficient (for further details on how this coefficients are updated refer to Appendix C.2).

Original: This newly defined distance metric can then be applied to calculate the probability of there existing an edge connecting latent features p p l q ij p
Revised: This newly defined distance metric can then be applied to calculate the probability of there existing an edge connecting latent features p p l q ij p

Original: [Equation]
Revised: [Equation]

Original: l q
Revised: l q

Original: [Equation]
Revised: [Equation]

Original: , f
Revised: , f

Original: [Equation]
Revised: [Equation]

Original: ˙˙ .
Revised: ˙˙ .

Original: Hence, E p l q p X p l q ; Θ p l q , T, k, d P q “ tp i, j i, 1 q , p i, j i, 2 q , ..., p i, j i,k q : i “ 1 , ..., N u .
Revised: Hence, E p l q p X p l q ; Θ p l q , T, k, d P q “ tp i, j i, 1 q , p i, j i, 2 q , ..., p i, j i,k q : i “ 1 , ..., N u .

Original: As discussed in Kazi et al.
Revised: As discussed in Kazi et al.

Original: (2022), the logarithms of the edge probabilities are used to update the dDGM.
Revised: (2022), the logarithms of the edge probabilities are used to update the dDGM.

Original: This is done by incorporating an additional term to the network loss function which will be dependent on
Revised: This is done by incorporating an additional term to the network loss function which will be dependent on

Original: [Equation]
Revised: [Equation]

Original: where the additional graph loss is given by L GL “ ř Ni “
Revised: where the additional graph loss is given by L GL “ ř Ni “

Original: [Equation]
Revised: [Equation]

Original: and δ p y i , ˆ y i q “ E p
Revised: and δ p y i , ˆ y i q “ E p

Original: ac
Revised: ac

Original: i q
Revised: i q

Original: ´ ac
Revised: ´ ac

Original: i is a reward function based on the expected accuracy of the model.
Revised: i is a reward function based on the expected accuracy of the model.

Original: The loss function to update the dDGM model, L GL , is identical to the original loss proposed by Kazi et al.
Revised: The loss function to update the dDGM model, L GL , is identical to the original loss proposed by Kazi et al.

Original: (2022), for a brief review one may refer to Appendix C.1.
Revised: (2022), for a brief review one may refer to Appendix C.1.

Original: Note that after passing the input x p l q i through the dDGM parameterized function f p l q Θ , the output f p l q Θ p x p l q i q “ ˆx p l ` 1 q i “ x p l q p i has dimension d E ` Σ n H j “ 1 d H j ` Σ n S k “ 1 d S k and must be subdivided into 1 ` n H ` n S subarrays for each of the component spaces.
Revised: Note that after passing the input x p l q i through the dDGM parameterized function f p l q Θ , the output f p l q Θ p x p l q i q “ ˆx p l ` 1 q i “ x p l q p i has dimension d E ` Σ n H j “ 1 d H j ` Σ n S k “ 1 d S k and must be subdivided into 1 ` n H ` n S subarrays for each of the component spaces.

Original: Each subarray must be appropriately mapped to its model space.
Revised: Each subarray must be appropriately mapped to its model space.

Original: Hence, the overline in f p l q Θ p x p l q i q in Equation 4.
Revised: Hence, the overline in f p l q Θ p x p l q i q in Equation 4.

Original: Finally, Figure 1 summarizes the method described in this section.
Revised: Finally, Figure 1 summarizes the method described in this section.

Original: Note that x p in Figure 1, would correspond to the concatenation of the origins of each model space composing the product manifold.
Revised: Note that x p in Figure 1, would correspond to the concatenation of the origins of each model space composing the product manifold.

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: The main objective of the experimental validation is to show that latent graph inference can benefit from using products of models spaces.
Revised: The main objective of the experimental validation is to show that latent graph inference can benefit from using products of models spaces.

Original: To do so, we compare the performance of the dDGM module when using single model spaces against Cartesian products of model spaces.
Revised: To do so, we compare the performance of the dDGM module when using single model spaces against Cartesian products of model spaces.

Original: The model spaces are denoted as: Euclidean (dDGM-E/dDGM ˚ -E, which is equivalent to the original architecture used by Kazi et al.
Revised: The model spaces are denoted as: Euclidean (dDGM-E/dDGM ˚ -E, which is equivalent to the original architecture used by Kazi et al.

Original: , hyperbolic (dDGM-H/dDGM ˚ -H), and spherical space (dDGM-S/dDGM ˚ -
Revised: , hyperbolic (dDGM-H/dDGM ˚ -H), and spherical space (dDGM-S/dDGM ˚ -

Original: S).
Revised: S).

Original: The asterisk sign in the model name denotes that the dDGM ˚ module is tasked with generating the latent graph without having access to the original adjacency matrix.
Revised: The asterisk sign in the model name denotes that the dDGM ˚ module is tasked with generating the latent graph without having access to the original adjacency matrix.

Original: dDGM models take as input X p 0 q and A p 0 q , whereas dDGM ˚ models only have access to X p 0 q .
Revised: dDGM models take as input X p 0 q and A p 0 q , whereas dDGM ˚ models only have access to X p 0 q .

Original: To refer to product manifolds, we simply append all the model spaces that compose the manifold to the name of the module, namely, the dDGM-SS module embedding space is a torus.
Revised: To refer to product manifolds, we simply append all the model spaces that compose the manifold to the name of the module, namely, the dDGM-SS module embedding space is a torus.

Original: In practice, we will use the same dimensionality but different curvature for each of the Cartesian components of the product manifolds.
Revised: In practice, we will use the same dimensionality but different curvature for each of the Cartesian components of the product manifolds.

Original: Lastly, if a GNN model uses the dDGM module, we name the diffusion layers and the latent graph inference module after.
Revised: Lastly, if a GNN model uses the dDGM module, we name the diffusion layers and the latent graph inference module after.

Original: For example, GCN-dDGM-E refers to a GCN that instead of using the original dataset graph for diffusion, incorporates latent graph inference to the network and uses the Euclidean plane as embedding space.
Revised: For example, GCN-dDGM-E refers to a GCN that instead of using the original dataset graph for diffusion, incorporates latent graph inference to the network and uses the Euclidean plane as embedding space.

Original: Also, note that we only use a single latent graph inference module per neural network, that is, networks diffuse information based on only one latent graph.
Revised: Also, note that we only use a single latent graph inference module per neural network, that is, networks diffuse information based on only one latent graph.

Original: This is in line with previous work (Kazi et al.
Revised: This is in line with previous work (Kazi et al.

Original: Additionally, in Appendix E.1, we investigate the effect of leveraging multiple latent graphs in the same network and conclude that in general it is better to use a single latent graph due to computational efficiency and diminishing returns.
Revised: Additionally, in Appendix E.1, we investigate the effect of leveraging multiple latent graphs in the same network and conclude that in general it is better to use a single latent graph due to computational efficiency and diminishing returns.

Original: Model descriptions for all experiments can be found in Appendix G.
Revised: Model descriptions for all experiments can be found in Appendix G.

Original: 4. H OMOPHILIC AND H ETEROPHILIC B ENCHMARK G RAPH D ATASETS
Revised: 4. H OMOPHILIC AND H ETEROPHILIC B ENCHMARK G RAPH D ATASETS

Original: We first focus on standard graph datasets widely discussed in the Geometric Deep Learning literature such as Cora, CiteSeer (Yang et al.
Revised: We first focus on standard graph datasets widely discussed in the Geometric Deep Learning literature such as Cora, CiteSeer (Yang et al.

Original: ; Lu & Getoor (2003); Sen et al.
Revised: ; Lu & Getoor (2003); Sen et al.

Original: (2008)), PubMed, Physics and CS (Shchur et al.
Revised: (2008)), PubMed, Physics and CS (Shchur et al.

Original: (2018)), which have high homophily levels ranging between 0.74 and 0.93.
Revised: (2018)), which have high homophily levels ranging between 0.74 and 0.93.

Original: We also present the results for several heterophilic datasets, which have homophily levels between 0. and 0.23.
Revised: We also present the results for several heterophilic datasets, which have homophily levels between 0. and 0.23.

Original: In particular, we work with Texas, Wisconsin, Squirrel, and Chameleon (Rozemberczki et al., 2021).
Revised: In particular, we work with Texas, Wisconsin, Squirrel, and Chameleon (Rozemberczki et al., 2021).

Original: Results of particular interest for this datasets are recorded in Table 2.
Revised: Results of particular interest for this datasets are recorded in Table 2.

Original: Additional experiments are available in Appendix E.2, E.3, and E.4, in which we perform an in depth exploration of different hyperparameters, compare dDGMs and dDGM ˚ s for all datasets, and try many product manifold combinations.
Revised: Additional experiments are available in Appendix E.2, E.3, and E.4, in which we perform an in depth exploration of different hyperparameters, compare dDGMs and dDGM ˚ s for all datasets, and try many product manifold combinations.

Original: Referring back to Table 2, we can see that product manifolds consistently outperform latent graph inference systems which only leverage a single model space for modeling the embedding space.
Revised: Referring back to Table 2, we can see that product manifolds consistently outperform latent graph inference systems which only leverage a single model space for modeling the embedding space.

Original: Also note, that unlike in the work by Kazi et al.
Revised: Also note, that unlike in the work by Kazi et al.

Original: (2022), we do find single hyperbolic model spaces to often outperform inference systems that use the Euclidean plane as embbeding space.
Revised: (2022), we do find single hyperbolic model spaces to often outperform inference systems that use the Euclidean plane as embbeding space.

Original: This shows that indeed mapping the Euclidean output features of the GNN layers to hyperbolic space using the exponential map before computing distances is of paramount importance (Kazi et al.
Revised: This shows that indeed mapping the Euclidean output features of the GNN layers to hyperbolic space using the exponential map before computing distances is of paramount importance (Kazi et al.

Original: (2022) ignored the exponential maps required to map features to the Poincar´e ball).
Revised: (2022) ignored the exponential maps required to map features to the Poincar´e ball).

Original: We have shown that the latent graph inference system enables GCN diffusion layers to achieve good performance for heterophilic datasets.
Revised: We have shown that the latent graph inference system enables GCN diffusion layers to achieve good performance for heterophilic datasets.

Original: We hypothesize that for this to be possible, it should be able to generate homophilic latent graphs, on which GCNs can easily diffuse.
Revised: We hypothesize that for this to be possible, it should be able to generate homophilic latent graphs, on which GCNs can easily diffuse.

Original: In Table 3 we display the homophily levels of the learned latent graphs, which corroborates our intuition.
Revised: In Table 3 we display the homophily levels of the learned latent graphs, which corroborates our intuition.

Original: As we can see from the results all models are able to generate latent graphs with higher homophily than those of the original dataset graphs.
Revised: As we can see from the results all models are able to generate latent graphs with higher homophily than those of the original dataset graphs.

Original: The latent graph inference system seems to find it easier to increase the homophily levels of smaller datasets, which is reasonable since there is less information to reorganize.
Revised: The latent graph inference system seems to find it easier to increase the homophily levels of smaller datasets, which is reasonable since there is less information to reorganize.

Original: There is a clear correlation between model performance in terms of accuracy (Table 2) and the homophily level that the dDGM ˚ modules are able to achieve for the latent graphs (Table 3).
Revised: There is a clear correlation between model performance in terms of accuracy (Table 2) and the homophily level that the dDGM ˚ modules are able to achieve for the latent graphs (Table 3).

Original: For example, in the case of Texas we achieve the highest homophily levels for the latent graphs of between 0 .
Revised: For example, in the case of Texas we achieve the highest homophily levels for the latent graphs of between 0 .

Original: 85 ˘ 0 .
Revised: 85 ˘ 0 .

Original: 02 and 0 .
Revised: 02 and 0 .

Original: 91 ˘ 0 . 03 , and also some of the highest accuracies ranging from
Revised: 91 ˘ 0 . 03 , and also some of the highest accuracies ranging from

Original: [Table]
Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 88 ˘ 9 .
Revised: 88 ˘ 9 .

Original: 95% to 81 .
Revised: 95% to 81 .

Original: 67 ˘ 7 .
Revised: 67 ˘ 7 .

Original: For Wisconsin the homophily level is lower than that for Texas, but this can be attributed to the fact that k “ 10 , inevitably creating more connections with nodes from other classes.
Revised: For Wisconsin the homophily level is lower than that for Texas, but this can be attributed to the fact that k “ 10 , inevitably creating more connections with nodes from other classes.

Original: Also, in Wisconsin there are two classes with substantially less nodes than the rest, meaning that a high accuracy can be achieved even if those are misclassified.
Revised: Also, in Wisconsin there are two classes with substantially less nodes than the rest, meaning that a high accuracy can be achieved even if those are misclassified.

Original: On the other hand, for Squirrel, although the latent graph inference system still manages to increase homophily from 0 .
Revised: On the other hand, for Squirrel, although the latent graph inference system still manages to increase homophily from 0 .

Original: 22 in the original graph to between 0 .
Revised: 22 in the original graph to between 0 .

Original: 27 ˘ 0 . 

Revised: 27 ˘ 0 .

Original: and 0 .
Revised: and 0 .

Original: 43 ˘ 0 .
Revised: 43 ˘ 0 .

Original: 03 in the latent graph, the increase is not big as compared to the other datasets and we can see how this also has an effect on performance.
Revised: 03 in the latent graph, the increase is not big as compared to the other datasets and we can see how this also has an effect on performance.

Original: In Table 2 the maximum accuracy for Squirrel is of 35 . 

Revised: In Table 2 the maximum accuracy for Squirrel is of 35 .

Original: ˘ 2 .
Revised: ˘ 2 .

Original: Note that this is still substantially better than using a MLP or a standard GCN, which obtain accuracies of 30 .
Revised: Note that this is still substantially better than using a MLP or a standard GCN, which obtain accuracies of 30 .

Original: 44 ˘ 2 .
Revised: 44 ˘ 2 .

Original: 55% and 24 .
Revised: 55% and 24 .

Original: 19 ˘ 2 .
Revised: 19 ˘ 2 .

Original: 56% , respectively.
Revised: 56% , respectively.

Original: The same discussion applies to Chameleon.
Revised: The same discussion applies to Chameleon.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Figure 2 displays how the graph connectivity is modified during the training process.
Revised: Figure 2 displays how the graph connectivity is modified during the training process.

Original: This shows that the inference system is able to dynamically learn an optimal connectivity structure for the latent graph based on the downstream task, and modify it accordingly during training.
Revised: This shows that the inference system is able to dynamically learn an optimal connectivity structure for the latent graph based on the downstream task, and modify it accordingly during training.

Original: Additional latent graph plots for the different datasets can be found in Appendix F. 
Revised: Additional latent graph plots for the different datasets can be found in Appendix F. [Figure]

Original: 4. R EAL -W ORLD A PPLICATIONS
Revised: R EAL -W ORLD A PPLICATIONS

Original: Next, we test the latent graph inference system on real-world applications: on the TadPole dataset (Marinescu et al.
Revised: Next, we test the latent graph inference system on real-world applications: on the TadPole dataset (Marinescu et al.

Original: (2020)), which was also discussed by Kazi et al.
Revised: (2020)), which was also discussed by Kazi et al.

Original: (2022), and the Aerothemodynamics dataset, which we have created for this work.
Revised: (2022), and the Aerothemodynamics dataset, which we have created for this work.

Original: The TadPole dataset contains information about brain images for different patients and the task is to classify each patient into three[Figure] classes: Normal Control , Alzheimer’s Disease and Mild Cognitive Impairment .
Revised: The TadPole dataset contains information about brain images for different patients and the task is to classify each patient into three classes: Normal Control , Alzheimer’s Disease and Mild Cognitive Impairment .

Original: On the other hand, the Aerothermodynamics dataset was inspired by recent research discussing potential applications of machine learning to aerospace engineering (Maheshwari et al.
Revised: On the other hand, the Aerothermodynamics dataset was inspired by recent research discussing potential applications of machine learning to aerospace engineering (Maheshwari et al.

Original: S´aez de Oc´ariz Borde et al.
Revised: ; S ´ aez de Oc ´ ariz Borde et al.

Original: (2021a;b)), and challenges networks to classify different regions of a shock wave around a rocket (refer to Appendix D for more information on the dataset).
Revised: (2021a;b)), and challenges networks to classify different regions of a shock wave around a rocket (refer to Appendix D for more information on the dataset).

Original: Note that since neither of these datasets have a graph structure (they are pointclouds), only the dDGM ˚ can be utilized in this case.
Revised: Note that since neither of these datasets have a graph structure (they are pointclouds), only the dDGM ˚ can be utilized in this case.

Original: Results are given in Table 4.
Revised: Results are given in Table 4.

Original: We use GAT diffusion layers and compare the performance using single model spaces and product manifolds.
Revised: We use GAT diffusion layers and compare the performance using single model spaces and product manifolds.

Original: Almost all models using the latent graph inference system outperform the performance of the MLP.
Revised: Almost all models using the latent graph inference system outperform the performance of the MLP.

Original: It is important to note that since the datasets do not provide an input graph it would not be possible to use GAT models without using dDGM ˚ modules.
Revised: It is important to note that since the datasets do not provide an input graph it would not be possible to use GAT models without using dDGM ˚ modules.

Original: Again, we find that using product manifolds to model the latent space of potentially complex real-world datasets proves beneficial and boosts model accuracy.
Revised: Again, we find that using product manifolds to model the latent space of potentially complex real-world datasets proves beneficial and boosts model accuracy.

Original: In principle, the Aerothermodynamics datasets classifies shock regions based on the flow absolute velocity into 4 regions as recorded in Table 4.
Revised: In principle, the Aerothermodynamics datasets classifies shock regions based on the flow absolute velocity into 4 regions as recorded in Table 4.

Original: However, we tested increasing the number of classes by further subdividing the flow into a total of 7 regions, as shown in Figure 3.
Revised: However, we tested increasing the number of classes by further subdividing the flow into a total of 7 regions, as shown in Figure 3.

Original: We found that interestingly, the latent graph inferred by the model does not only cluster nodes with similar labels together, but it actually organizes the latent graph in order based on the absolute velocities (which are not explicitly given to the model: the velocities are used to create the labels but values are not provided to the model as input).
Revised: We found that interestingly, the latent graph inferred by the model does not only cluster nodes with similar labels together, but it actually organizes the latent graph in order based on the absolute velocities (which are not explicitly given to the model: the velocities are used to create the labels but values are not provided to the model as input).

Original: This suggests that the graph generation system is organizing the latent graph based on some inferred high level understanding of the physics of the problem.
Revised: This suggests that the graph generation system is organizing the latent graph based on some inferred high level understanding of the physics of the problem.

Original: Additional latent graphs for these datasets are provided in Appendix F.2.[Figure]
Revised: Additional latent graphs for these datasets are provided in Appendix F.2.

Original: 4. S CALING TO L ARGE G RAPHS
Revised: 4. S CALING TO L ARGE G RAPHS

Original: All datasets considered so far are relatively small.
Revised: All datasets considered so far are relatively small.

Original: In this section we work with datasets from the
Revised: In this section we work with datasets from the

Original: Open Graph Benchmark (OGB) which contain large-scale graphs and require models to perform realistic out-of-distribution generalization.
Revised: Open Graph Benchmark (OGB) which contain large-scale graphs and require models to perform realistic out-of-distribution generalization.

Original: In particular, we use the OGB-Arxiv and the OGBProducts datasets.
Revised: In particular, we use the OGB-Arxiv and the OGBProducts datasets.

Original: As discussed in Appendix C.3.3, for training on these datasets we use graph subsampling techniques, and Graph Attention Network version 2 (GATv2) diffusion layers (Brody et al.
Revised: As discussed in Appendix C.3.3, for training on these datasets we use graph subsampling techniques, and Graph Attention Network version 2 (GATv2) diffusion layers (Brody et al.

Original: ; since we do not expect overfitting we can use more expressive layers.
Revised: ; since we do not expect overfitting we can use more expressive layers.

Original: OGB-Arxiv and OGB-Products have a total of 40 and 47 classes, respectively.
Revised: OGB-Arxiv and OGB-Products have a total of 40 and 47 classes, respectively.

Original: Previous datasets only considered  multi-class classification for between 3 to 15 classes.
Revised: Previous datasets only considered [Figure] multi-class classification for between 3 to 15 classes.

Original: This, added to the fact that the datasets have orders of magnitude more nodes and edges, makes the problems in this section considerably more challenging.
Revised: This, added to the fact that the datasets have orders of magnitude more nodes and edges [Figure] makes the problems in this section considerably more challenging.

Original: For OGB-Arxiv using a MLP and a GATv2 model without leveraging latent graph inference we obtain accuracies of 63 .
Revised: For OGB-Arxiv using a MLP and a GATv2 model without leveraging latent graph inference we obtain accuracies of 63 .

Original: 49 ˘ 0 .
Revised: 49 ˘ 0 .

Original: 15% and 61 .
Revised: 15% and 61 .

Original: 93 ˘ 1 .
Revised: 93 ˘ 1 .

Original: 62% , respectively.
Revised: 62% , respectively.

Original: The best model with latent graph inference, GATv2-dDGM ˚ -EHS, achieves an accurary of 65 . 06 ˘ 0 .
Revised: The best model with latent graph inference, GATv2-dDGM ˚ -EHS, achieves an accurary of 65 . 06 ˘ 0 .

Original: For
Revised: For

Original: OGB-Products, the MLP and a GATv2 results are 66 .
Revised: OGB-Products, the MLP and a GATv2 results are 66 .

Original: 05 ˘ 0 .
Revised: 05 ˘ 0 .

Original: 20% and 62 .
Revised: 20% and 62 .

Original: 02 ˘ 2 .
Revised: 02 ˘ 2 .

Original: 60% , and for the best model, GATv2-dDGM-E, we record an accuracy of 66 .
Revised: 60% , and for the best model, GATv2-dDGM-E, we record an accuracy of 66 .

Original: 59 ˘ 0 .
Revised: 59 ˘ 0 .

Original: From the results (more in Appendix E.5), we conclude that latent graph inference is still beneficial for this larger datasets but there is substantial room for improvement.
Revised: From the results (more in Appendix E.5), we conclude that latent graph inference is still beneficial for this larger datasets but there is substantial room for improvement.

Original: Graph subsampling interferes with embedding space learning.
Revised: Graph subsampling interferes with embedding space learning.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: In this work we have incorporated Riemannian geometry to the dDGM latent graph inference module by Kazi et al.
Revised: In this work we have incorporated Riemannian geometry to the dDGM latent graph inference module by Kazi et al.

Original: First, we have shown how to work with manifolds of constant arbitrary curvature, both positive and negative.
Revised: First, we have shown how to work with manifolds of constant arbitrary curvature, both positive and negative.

Original: Next, we have leveraged product manifolds of model spaces and their convenient mathematical properties to enable the dDGM module to generate a more complex homogeneous manifold with varying curvature which can better encode the latent data, while learning the curvature of each model space composing the product manifold during training.
Revised: Next, we have leveraged product manifolds of model spaces and their convenient mathematical properties to enable the dDGM module to generate a more complex homogeneous manifold with varying curvature which can better encode the latent data, while learning the curvature of each model space composing the product manifold during training.

Original: We have evaluated our method on many and diverse datasets, and we have shown that using product manifolds to model the embedding space for the latent graph gives enhanced downstream performance as compared to using single model spaces of constant curvature.
Revised: We have evaluated our method on many and diverse datasets, and we have shown that using product manifolds to model the embedding space for the latent graph gives enhanced downstream performance as compared to using single model spaces of constant curvature.

Original: The inference system has been tested on both homophilic and heterophilic benchmarks.
Revised: The inference system has been tested on both homophilic and heterophilic benchmarks.

Original: In particular, we have found that using optimized latent graphs, diffusion layers like GCNs are able to successfully operate on datasets with low homophily levels.
Revised: In particular, we have found that using optimized latent graphs, diffusion layers like GCNs are able to successfully operate on datasets with low homophily levels.

Original: Additionally, we have tested and proven the applicability of our method to large-scale graphs.
Revised: Additionally, we have tested and proven the applicability of our method to large-scale graphs.

Original: Lastly, we have shown the benefits of applying this procedure in real-world problems such as brain imaging based data and aerospace engineering problems.
Revised: Lastly, we have shown the benefits of applying this procedure in real-world problems such as brain imaging based data and aerospace engineering problems.

Original: All experiments discussed in the main text are concerned with transductive learning; however, the method is also applicable to inductive learning, see Appendix E.6.
Revised: All experiments discussed in the main text are concerned with transductive learning; however, the method is also applicable to inductive learning, see Appendix E.6.

Original: The product manifold embedding space approach has provided a computationally tractable way of generating more complex homogenous manifolds for the latent features’ embedding space.
Revised: The product manifold embedding space approach has provided a computationally tractable way of generating more complex homogenous manifolds for the latent features’ embedding space.

Original: Furthermore, the curvature of the product components is learned rather than it being a fixed hyperparameter, which allows for greater flexibility.
Revised: Furthermore, the curvature of the product components is learned rather than it being a fixed hyperparameter, which allows for greater flexibility.

Original: However, the number of model spaces to generate the product manifold must be specified before training.
Revised: However, the number of model spaces to generate the product manifold must be specified before training.

Original: It would be interesting to devise an approach for the network to independently add more model spaces to the product manifold when needed.
Revised: It would be interesting to devise an approach for the network to independently add more model spaces to the product manifold when needed.

Original: Also, we are restricting our approach to product manifolds based on model spaces of constant curvature due to their suitable mathematical properties.
Revised: Also, we are restricting our approach to product manifolds based on model spaces of constant curvature due to their suitable mathematical properties.

Original: Such product manifolds do not cover all possible arbitrary manifolds in which the latent data could be encoded and hence, there could still be, mathematically speaking, more optimal manifolds to represent the data.
Revised: Such product manifolds do not cover all possible arbitrary manifolds in which the latent data could be encoded and hence, there could still be, mathematically speaking, more optimal manifolds to represent the data.

Original: It is worth exploring whether approaches to generate even more diverse and computationally tractable manifolds would be possible.
Revised: It is worth exploring whether approaches to generate even more diverse and computationally tractable manifolds would be possible.

Original: Future Work Lastly, there are a few limitations intrinsic to the dDGM module, irrespective of the product manifold embedding approach introduced in this work.
Revised: Work Lastly, there are a few limitations intrinsic to the dDGM module, irrespective of the product manifold embedding approach introduced in this work.

Original: Firstly, although utilizing symbolic matrices can help computational efficiency (Appendix C.3), the method still has quadratic complexity.
Revised: Firstly, although utilizing symbolic matrices can help computational efficiency (Appendix C.3), the method still has quadratic complexity.

Original: Kazi et al.
Revised: Kazi et al.

Original: (2022) proposed computing probabilities in a neighborhood of the node and using tree-based algorithms to reduce it to O p n log n q .
Revised: (2022) proposed computing probabilities in a neighborhood of the node and using tree-based algorithms to reduce it to O p n log n q .

Original: Moreover, the Gumbel Top-k sampling approach restricts the average node degree of the latent graph and requires manually adjusting the k value through testing.
Revised: Moreover, the Gumbel Top-k sampling approach restricts the average node degree of the latent graph and requires manually adjusting the k value through testing.

Original: A possible solution could be to use a distance based sparse threshold approach in which an unweighted edge is created between two nodes if they are within a threshold distance of each other in latent space.
Revised: A possible solution could be to use a distance based sparse threshold approach in which an unweighted edge is created between two nodes if they are within a threshold distance of each other in latent space.

Original: This is similar to the Gumbel Top-k trick, but instead of choosing a fixed number of closest neighbors, we connect all nodes within a distance.
Revised: This is similar to the Gumbel Top-k trick, but instead of choosing a fixed number of closest neighbors, we connect all nodes within a distance.

Original: This could help better capture the heterogeneity of the graph.
Revised: This could help better capture the heterogeneity of the graph.

Original: However, we actually tested this approach and found it quite unstable.
Revised: However, we actually tested this approach and found it quite unstable.

Original: Note that although we do not have the k parameter anymore, we must still choose a threshold distance.
Revised: Note that although we do not have the k parameter anymore, we must still choose a threshold distance.

Original: Another avenue to help improve computational complexity and facilitate working with large-scale graphs would be to use a hierarchical perspective.
Revised: Another avenue to help improve computational complexity and facilitate working with large-scale graphs would be to use a hierarchical perspective.

Original: Inspired by brain interneurons (Freund & Buzs´aki (1996)), we could introduce fictitious connector inducing nodes in different regions of the graph, use those nodes to summarize different regions of large graphs, and apply the kNN algorithm or the Gumbel Top-k trick to the fictitious connector inducing nodes.
Revised: Inspired by brain interneurons (Freund & Buzs´aki (1996)), we could introduce fictitious connector inducing nodes in different regions of the graph, use those nodes to summarize different regions of large graphs, and apply the kNN algorithm or the Gumbel Top-k trick to the fictitious connector inducing nodes.

Original: This way the computational complexity would still be quadratic, but proportional to the number of interconnectors.
Revised: This way the computational complexity would still be quadratic, but proportional to the number of interconnectors.

Original: Similar techniques have been applied to Gaussian Processes (Galy-Fajou & Opper (2021); Wu et al.
Revised: Similar techniques have been applied to Gaussian Processes (Galy-Fajou & Opper (2021); Wu et al.

Original: (2021)) and
Revised: (2021)) and

Original: Set Transformers (Lee et al.
Revised: Set Transformers (Lee et al.

Original: A R IEMANNIAN M ANIFOLDS
Revised: A R IEMANNIAN M ANIFOLDS

Original: In Riemannian geometry, we define a Riemannian manifold or Riemannian space p M , g q as a real and differentiable manifold M in which each tangent space has an associated inner product g , that is, a Riemannian metric, which must vary smoothly when considering points in the manifold.
Revised: In Riemannian geometry, we define a Riemannian manifold or Riemannian space p M , g q as a real and differentiable manifold M in which each tangent space has an associated inner product g , that is, a Riemannian metric, which must vary smoothly when considering points in the manifold.

Original: The
Revised: The

Original: Riemannian manifold M Ď R N (lives in the ambient space R N ) is a collection of real vectors, and is locally similar to a linear space.
Revised: Riemannian manifold M Ď R N (lives in the ambient space R N ) is a collection of real vectors, and is locally similar to a linear space.

Original: The Riemannian metric generalizes inner products for Riemannian manifolds.
Revised: The Riemannian metric generalizes inner products for Riemannian manifolds.

Original: It also allows to define geometric notions on a Riemannian manifold such as lengths of curves, curvature, and angles to name a few.
Revised: It also allows to define geometric notions on a Riemannian manifold such as lengths of curves, curvature, and angles to name a few.

Original: If the point x p P M , then we can denote the tangent space at x p as T x p M , which has the same dimensionality as M .
Revised: If the point x p P M , then we can denote the tangent space at x p as T x p M , which has the same dimensionality as M .

Original: T x p M is the collection of all tangent vectors at x p .
Revised: T x p M is the collection of all tangent vectors at x p .

Original: Moreover, g x p : T x p M ˆ T x p M Ñ R is given by a positivedefinite inner product in the tangent space and depends smoothly on x p .
Revised: Moreover, g x p : T x p M ˆ T x p M Ñ R is given by a positivedefinite inner product in the tangent space and depends smoothly on x p .

Original: A geodesic represents the shortest smooth path between two points in a Riemannian manifold and generalizes the notion of a straight line in Euclidean space.
Revised: A geodesic represents the shortest smooth path between two points in a Riemannian manifold and generalizes the notion of a straight line in Euclidean space.

Original: The length of a smooth continuously differentiable curve γ : t Ñ γ p t q P M , t P r 0 , 1 s is given by
Revised: The length of a smooth continuously differentiable curve γ : t Ñ γ p t q P M , t P r 0 , 1 s is given by

Original: [Equation]
Revised: [Equation]

Original: Note that L p γ q is unchanged by a monotone reparametrization.
Revised: Note that L p γ q is unchanged by a monotone reparametrization.

Original: The geodesic distance between two points x p i , x p j P M is defined as the infimum (greatest lower bound) of the length taken over all piecewise continuously differentiable curves, such that
Revised: The geodesic distance between two points x p i , x p j P M is defined as the infimum (greatest lower bound) of the length taken over all piecewise continuously differentiable curves, such that

Original: [Equation]
Revised: [Equation]

Original: The norm of a tangent vector v P T x p M is given by
Revised: The norm of a tangent vector v P T x p M is given by

Original: [Equation]
Revised: [Equation]

Original: Moving from a point x p P M with initial constant velocity v P T x p M is formalized by the exponential map
Revised: Moving from a point x p P M with initial constant velocity v P T x p M is formalized by the exponential map

Original: [Equation]
Revised: [Equation]

Original: which gives the position of the geodesic at t “ 1 so that
Revised: which gives the position of the geodesic at t “ 1 so that

Original: [Equation]
Revised: [Equation]

Original: There is a unique unit speed geodesic γ which satisfies γ p 0 q “ x p and γ 1 p 0 q “ v .
Revised: There is a unique unit speed geodesic γ which satisfies γ p 0 q “ x p and γ 1 p 0 q “ v .

Original: On the other hand, and less relevant to the work at hand, the logarithmic map is the inverse of the exponential map
Revised: On the other hand, and less relevant to the work at hand, the logarithmic map is the inverse of the exponential map

Original: [Equation]
Revised: [Equation]

Original: In geodesically complete Riemannian manifolds, both the exponential and logarithm maps are welldefined (Needham (1997)).
Revised: In geodesically complete Riemannian manifolds, both the exponential and logarithm maps are welldefined (Needham (1997)).

Original: In general, it can be impossible to find a solution for the geodesic between two points for an arbitrary manifold.
Revised: In general, it can be impossible to find a solution for the geodesic between two points for an arbitrary manifold.

Original: As discussed in the main text, we want to be able to move beyond constant curvature spaces and compute the similarity measure φ for latent features which may reside in a more general and learnable manifold.
Revised: As discussed in the main text, we want to be able to move beyond constant curvature spaces and compute the similarity measure φ for latent features which may reside in a more general and learnable manifold.

Original: This will enable us to more accurately model data which may comprise varying structure, beyond that which can be represented in Euclidean space, and also beyond hierarchical or cyclical data which can be linked to constant curvature hyperbolic and spherical spaces, respectively.
Revised: This will enable us to more accurately model data which may comprise varying structure, beyond that which can be represented in Euclidean space, and also beyond hierarchical or cyclical data which can be linked to constant curvature hyperbolic and spherical spaces, respectively.

Original: That is, by varying structure we refer to data which may present different underlying patterns in different regions of space.
Revised: That is, by varying structure we refer to data which may present different underlying patterns in different regions of space.

Original: Generating more complex manifolds we intend to minimize the distortion incurred in the data and to represent the points in a more suitable manifold for the downstream task.
Revised: Generating more complex manifolds we intend to minimize the distortion incurred in the data and to represent the points in a more suitable manifold for the downstream task.

Original: However, we must still be able to map the Euclidean output features of the network to our learnable manifold and to compute distances between points.
Revised: However, we must still be able to map the Euclidean output features of the network to our learnable manifold and to compute distances between points.

Original: To generate a more complex and learnable manifold while still having a closed form solution for the exponential map and geodesics, we will introduce a product manifold embedding space composed of multiple copies of simple model spaces with constant curvature.
Revised: To generate a more complex and learnable manifold while still having a closed form solution for the exponential map and geodesics, we will introduce a product manifold embedding space composed of multiple copies of simple model spaces with constant curvature.

Original: Although product manifolds based on constant curvature models still fall under the homogeneous manifold category (Kowalski et al.
Revised: Although product manifolds based on constant curvature models still fall under the homogeneous manifold category (Kowalski et al.

Original: , they allow to model more complex embedding spaces for the dDGM module than those that can be represented using only constant curvature homogeneous spaces.
Revised: , they allow to model more complex embedding spaces for the dDGM module than those that can be represented using only constant curvature homogeneous spaces.

Original: For example, the standard torus, which can be obtained by multiplying two spheres, is a homogeneous manifold.
Revised: For example, the standard torus, which can be obtained by multiplying two spheres, is a homogeneous manifold.

Original: Nevertheless, some points on the manifold have positive Gaussian curvature (outer part of the torus, elliptic points), some have negative (inner part, hyperbolic points), and others have zero (parabolic points).
Revised: Nevertheless, some points on the manifold have positive Gaussian curvature (outer part of the torus, elliptic points), some have negative (inner part, hyperbolic points), and others have zero (parabolic points).

Original: This is because for the torus the Euler characteristic (Beltramo et al.
Revised: This is because for the torus the Euler characteristic (Beltramo et al.

Original: (2021)) is zero so it must always have regions of negative Gaussian curvature.
Revised: (2021)) is zero so it must always have regions of negative Gaussian curvature.

Original: This is to counterbalance the regions of positive curvature guaranteed in Hilbert’s theorem.
Revised: This is to counterbalance the regions of positive curvature guaranteed in Hilbert’s theorem.

Original: The main point is that using product manifolds of constant curvature spaces we can generate manifolds with regions with different Gaussian curvature that can help us better represent data structures which may not be purely Euclidean, hyperbolic, or spherical.
Revised: The main point is that using product manifolds of constant curvature spaces we can generate manifolds with regions with different Gaussian curvature that can help us better represent data structures which may not be purely Euclidean, hyperbolic, or spherical.

Original: For all points on the manifold, we can define a normal vector that is at right angles to the surface.
Revised: For all points on the manifold, we can define a normal vector that is at right angles to the surface.

Original: By generating intersections between normal planes (that contain the normal vector) and the surface we can compute normal sections.
Revised: By generating intersections between normal planes (that contain the normal vector) and the surface we can compute normal sections.

Original: In general, different normal sections will have different curvatures.
Revised: In general, different normal sections will have different curvatures.

Original: We refer to κ 1 and κ 2 as the principal curvatures.
Revised: We refer to κ 1 and κ 2 as the principal curvatures.

Original: They correspond to the maximum and minimum values that the curvature can take at a given point (K ¨ uhnel (2005)).
Revised: They correspond to the maximum and minimum values that the curvature can take at a given point (K ¨ uhnel (2005)).

Original: The Gaussian curvature K is the product of the two
Revised: The Gaussian curvature K is the product of the two

Original: [Equation]
Revised: [Equation]

Original: A Riemannian manifold is said to have constant Gaussian curvature K if sec p P q “ K for all two-dimensional linear subspaces P Ă T x p M and for all x p P M .
Revised: A Riemannian manifold is said to have constant Gaussian curvature K if sec p P q “ K for all two-dimensional linear subspaces P Ă T x p M and for all x p P M .

Original: Manifolds can be classified into three classes depending on their curvature: flat space, positively curved space, and negatively curved space.
Revised: Manifolds can be classified into three classes depending on their curvature: flat space, positively curved space, and negatively curved space.

Original: B F URTHER D ISCUSSION ON C URVATURE L EARNING
Revised: B F URTHER D ISCUSSION ON C URVATURE L EARNING

Original: Next, we aim to provide a more detailed explanation on the topic of curvature learning for product manifolds and the implemented procedure using learnable distance-metric-scaling coefficients.
Revised: Next, we aim to provide a more detailed explanation on the topic of curvature learning for product manifolds and the implemented procedure using learnable distance-metric-scaling coefficients.

Original: Let us discuss product manifolds which are solely generated based on Cartesian products of the same model spaces, such as
Revised: Let us discuss product manifolds which are solely generated based on Cartesian products of the same model spaces, such as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which uses Cartesian products of hyperboloids.
Revised: which uses Cartesian products of hyperboloids.

Original: Likewise, taking Cartesian products of hyperspheres we would obtain
Revised: Likewise, taking Cartesian products of hyperspheres we would obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For these cases, although d H d H j
Revised: For these cases, although d H d H j

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: ¯ are in practice computed all using hyperboloids and hyperspheres with the same fixed curvature K H j “ ´ 1 , @
Revised: ¯ are in practice computed all using hyperboloids and hyperspheres with the same fixed curvature K H j “ ´ 1 , @

Original: j , and K S k “ 1 , @
Revised: j , and K S k “ 1 , @

Original: k , the scaling coefficients control the curvature of the spaces individually,
Revised: k , the scaling coefficients control the curvature of the spaces individually,

Original: [Equation]
Revised: [Equation]

Original: and,
Revised: and,

Original: [Equation]
Revised: [Equation]

Original: That is, α H j and α S k scale the distances generated based on unit hyperboloids and hyperspheres, respectively.
Revised: That is, α H j and α S k scale the distances generated based on unit hyperboloids and hyperspheres, respectively.

Original: Using the scaled metrics, we are effectively still computing Cartesian products of hyperboloids and hyperspheres with different curvatures, so that
Revised: Using the scaled metrics, we are effectively still computing Cartesian products of hyperboloids and hyperspheres with different curvatures, so that

Original: [Equation]
Revised: [Equation]

Original: This guarantees we are retrieving equivalent values to those which would be generated by model spaces with different curvatures.
Revised: This guarantees we are retrieving equivalent values to those which would be generated by model spaces with different curvatures.

Original: This is done to avoid backpropagating through operators such as exponential maps and closed form solutions for distances in the different model spaces.
Revised: This is done to avoid backpropagating through operators such as exponential maps and closed form solutions for distances in the different model spaces.

Original: Considering again an arbitrary product manifold of all model spaces as in Equation 2, we can control the curvature through the following derivatives
Revised: Considering again an arbitrary product manifold of all model spaces as in Equation 2, we can control the curvature through the following derivatives

Original: [Equation]
Revised: [Equation]

Original: for the hyperboloid terms, and
Revised: for the hyperboloid terms, and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for the hyperspheres.
Revised: for the hyperspheres.

Original: In the case of Euclidean space the curvature is always K E “ 0 , so there is no need to learn it.
Revised: In the case of Euclidean space the curvature is always K E “ 0 , so there is no need to learn it.

Original: Also, using a Cartesian product of Euclidean spaces would be equivalent to using a single Euclidean space of greater dimensionality
Revised: Also, using a Cartesian product of Euclidean spaces would be equivalent to using a single Euclidean space of greater dimensionality

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where K E i “ K E “ 0 , @ i , and d E “ ř n E i d E i .
Revised: where K E i “ K E “ 0 , @ i , and d E “ ř n E i d E i .

Original: Hence, in Equation 2 we used a single Euclidean space.
Revised: Hence, in Equation 2 we used a single Euclidean space.

Original: The behavior described in Equation 20 can be better appreciated by comparing distance d P E obtained using P E and d E d E K E from E d E K E :
Revised: The behavior described in Equation 20 can be better appreciated by comparing distance d P E obtained using P E and d E d E K E from E d E K E :

Original: [Equation]
Revised: [Equation]

Original: Considering that the equation for independent distances in each Euclidean space is
Revised: Considering that the equation for independent distances in each Euclidean space is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: we obtain,
Revised: we obtain,

Original: [Equation]
Revised: [Equation]

Original: which is analogous to taking the second power of the difference of two points in another Euclidean space with more dimensions, so that
Revised: which is analogous to taking the second power of the difference of two points in another Euclidean space with more dimensions, so that

Original: [Equation]
Revised: [Equation]

Original: This behavior is only applicable to Euclidean space (Gu et al.
Revised: This behavior is only applicable to Euclidean space (Gu et al.

Original: (2019)), when multiplying other model spaces
Revised: (2019)), when multiplying other model spaces

Original: [Equation]
Revised: [Equation]

Original: even when the curvature is the same for all hyperboloids and hyperspheres, K H j “ K H , @
Revised: even when the curvature is the same for all hyperboloids and hyperspheres, K H j “ K H , @

Original: j and
Revised: j and

Original: K S k “ K S , @ k .
Revised: K S k “ K S , @ k .

Original: For example, multiplying two hyperspheres will result in a hypertorus, not a higherdimensional hypersphere.
Revised: For example, multiplying two hyperspheres will result in a hypertorus, not a higherdimensional hypersphere.

Original: As a final remark, note that any loss function which is dependent on the distance function induced by the Riemannian metric of a Riemannian manifold is locally smooth on M (where M could be a product manifold P ) and can be optimized by first order methods.
Revised: As a final remark, note that any loss function which is dependent on the distance function induced by the Riemannian metric of a Riemannian manifold is locally smooth on M (where M could be a product manifold P ) and can be optimized by first order methods.

Original: C T RAINING P ROCEDURE
Revised: C T RAINING P ROCEDURE

Original: In this appendix we discuss key concepts to train the dDGM module.
Revised: In this appendix we discuss key concepts to train the dDGM module.

Original: We detail how to backpropagate through the discrete sampling method and introduce an additional loss term to make this possible.
Revised: We detail how to backpropagate through the discrete sampling method and introduce an additional loss term to make this possible.

Original: We also we provide additional implementation details for updating learnable distancemetric-scaling coefficients and dealing with distance functions during training.
Revised: We also we provide additional implementation details for updating learnable distancemetric-scaling coefficients and dealing with distance functions during training.

Original: Lastly, we describe the two approaches used in this work to make training computationally tractable for larger graphs: symbolic handling of distance metrics and graph subsampling.
Revised: Lastly, we describe the two approaches used in this work to make training computationally tractable for larger graphs: symbolic handling of distance metrics and graph subsampling.

Original: C. B ACKPROPAGATION THROUGH THE D DGM
Revised: C. B ACKPROPAGATION THROUGH THE D DGM

Original: The baseline node feature learning part of the architecture is optimized based on the downstream task loss: for classification we use the cross-entropy loss and for regression the mean squared error loss.
Revised: The baseline node feature learning part of the architecture is optimized based on the downstream task loss: for classification we use the cross-entropy loss and for regression the mean squared error loss.

Original: Nevertheless, we must also update the graph learning dDGM parameters.
Revised: Nevertheless, we must also update the graph learning dDGM parameters.

Original: To do so, we follow the approach proposed by Kazi et al.
Revised: To do so, we follow the approach proposed by Kazi et al.

Original: and apply a compound loss that rewards edges involved in a correct classification and penalizes edges which result in misclassification.
Revised: and apply a compound loss that rewards edges involved in a correct classification and penalizes edges which result in misclassification.

Original: We define the reward function,
Revised: We define the reward function,

Original: [Equation]
Revised: [Equation]

Original: (28) as the difference between the average accuracy of the i th sample and the current prediction accuracy, where y i and ˆ y i are the predicted and true labels, and
Revised: (28) as the difference between the average accuracy of the i th sample and the current prediction accuracy, where y i and ˆ y i are the predicted and true labels, and

Original: ac i “ 1 if y i
Revised: ac i “ 1 if y i

Original: “ ˆ y i or 0 otherwise.
Revised: “ ˆ y i or 0 otherwise.

Original: Based on δ p y i , ˆ y i q we obtain the loss employed to update the graph learning module
Revised: Based on δ p y i , ˆ y i q we obtain the loss employed to update the graph learning module

Original: [Equation]
Revised: [Equation]

Original: L GL ’s gradient approximates the gradient of the expectation
Revised: L GL ’s gradient approximates the gradient of the expectation

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: with respect to the parameters of the graphs in all the layer θ GL .
Revised: with respect to the parameters of the graphs in all the layer θ GL .

Original: So that,
Revised: So that,

Original: [Equation]
Revised: [Equation]

Original: The expectation E p ac i q p t q is calculated based on
Revised: The expectation E p ac i q p t q is calculated based on

Original: [Equation]
Revised: [Equation]

Original: with β “ 0 . 9 and E p ac i q p t “ 0 q “ 0 . 5 .
Revised: with β “ 0 . 9 and E p ac i q p t “ 0 q “ 0 . 5 .

Original: For regression we use the R2 score instead of the accuracy.
Revised: For regression we use the R2 score instead of the accuracy.

Original: C. U PDATING L EARNABLE D ISTANCE - METRIC - SCALING C OEFFICIENTS
Revised: C. U PDATING L EARNABLE D ISTANCE - METRIC - SCALING C OEFFICIENTS

Original: The scaling metrics are learnable, so that we can indirectly adjust the curvature of each model space without having to backpropagate through the exponential map functions and the formulas for the distances.
Revised: The scaling metrics are learnable, so that we can indirectly adjust the curvature of each model space without having to backpropagate through the exponential map functions and the formulas for the distances.

Original: If we simply take the derivative of the graph loss and update the distance-metric-scaling coefficients during training α p t q “ α p t ´ 1 q ´ lr B L GL B α p t ´ 1 q , ( lr being the learning rate) this can result in negative values for the coefficients that multiply the distances in different model spaces, which would be mathematically incorrect since distances are by definition positive or zero.
Revised: If we simply take the derivative of the graph loss and update the distance-metric-scaling coefficients during training α p t q “ α p t ´ 1 q ´ lr B L GL B α p t ´ 1 q , ( lr being the learning rate) this can result in negative values for the coefficients that multiply the distances in different model spaces, which would be mathematically incorrect since distances are by definition positive or zero.

Original: To solve this issue we learn ˜ α , instead of α .
Revised: To solve this issue we learn ˜ α , instead of α .

Original: The two are related by the following equation, α “ S p ˜ α q , where
Revised: The two are related by the following equation, α “ S p ˜ α q , where

Original: S is the sigmoid function.
Revised: S is the sigmoid function.

Original: So that, α p t q “ S p ˜ α p t q q “ S ` ˜ α p t ´ 1 q ´ lr B L GL B ˜ α p t ´ 1 q ˘ , which means that 

Revised: So that, α p t q “ S p ˜ α p t q q “ S ` ˜ α p t ´ 1 q ´ lr B L GL B ˜ α p t ´ 1 q ˘ , which means that

Original: ă α p t q ă 1 .
Revised: ă α p t q ă 1 .

Original: Using ReLU instead of S could allow α p t q to take arbitrarily large positive values, but we would have gradient problems if ˜ α becomes negative.
Revised: Using ReLU instead of S could allow α p t q to take arbitrarily large positive values, but we would have gradient problems if ˜ α becomes negative.

Original: Given that using the sigmoid function bounds the maximum value that the scaling metrics can take, we must multiply the Euclidean plane distance by its own scaling coefficient.
Revised: Given that using the sigmoid function bounds the maximum value that the scaling metrics can take, we must multiply the Euclidean plane distance by its own scaling coefficient.

Original: Since the Gumbel Top-k trick selects the closest points to generate unweighted edges, rather than storing the actual geodesics, if we scale the Euclidean plane contribution down when necessary, this would equate to having other model spaces with much bigger curvature than that which is actually possible due to the scaling coefficient being now bounded by zero and one.
Revised: Since the Gumbel Top-k trick selects the closest points to generate unweighted edges, rather than storing the actual geodesics, if we scale the Euclidean plane contribution down when necessary, this would equate to having other model spaces with much bigger curvature than that which is actually possible due to the scaling coefficient being now bounded by zero and one.

Original: For example, if we were to have a product manifold based on the Cartesian product of the Euclidean plane and a hypersphere, if the model learns a scaling metric close to zero for the Euclidean plane geodesics, this would be equivalent to having a hypersphere with a really large curvature since edges would only be generated based on the distances calculated on the hypersphere.
Revised: For example, if we were to have a product manifold based on the Cartesian product of the Euclidean plane and a hypersphere, if the model learns a scaling metric close to zero for the Euclidean plane geodesics, this would be equivalent to having a hypersphere with a really large curvature since edges would only be generated based on the distances calculated on the hypersphere.

Original: Finally, note that in practice we use the Adam optimizer instead of simple gradient descent for training.
Revised: Finally, note that in practice we use the Adam optimizer instead of simple gradient descent for training.

Original: C. C OMPUTATIONAL E FFICIENCY
Revised: C. C OMPUTATIONAL E FFICIENCY

Original: In this section we discuss some of the implementation techniques used to make computation more efficient.
Revised: In this section we discuss some of the implementation techniques used to make computation more efficient.

Original: We cover two main topics: symbolic handling of distance metrics and graph subsampling.
Revised: We cover two main topics: symbolic handling of distance metrics and graph subsampling.

Original: One of the main computational limitations of the approach described in this work is that we must compute distances between all points to generate the latent graph.
Revised: One of the main computational limitations of the approach described in this work is that we must compute distances between all points to generate the latent graph.

Original: Although the discrete graph sampling method used by dDGM is more computationally efficient than its continuous counterpart, cDGM — because it generates sparse graphs that make convolutional operators lighter — we quickly run into memory problems for graph datasets with O p 10 4 q nodes.
Revised: Although the discrete graph sampling method used by dDGM is more computationally efficient than its continuous counterpart, cDGM — because it generates sparse graphs that make convolutional operators lighter — we quickly run into memory problems for graph datasets with O p 10 4 q nodes.

Original: Starting from a pointcloud, we must compute the distances between all points to determine whether a connection should be established.
Revised: Starting from a pointcloud, we must compute the distances between all points to determine whether a connection should be established.

Original: This is problematic, since the computational complexity scales quadratically as a function of the number of nodes in the graph, which can rapidly become intractable as we increase the size of our graph.
Revised: This is problematic, since the computational complexity scales quadratically as a function of the number of nodes in the graph, which can rapidly become intractable as we increase the size of our graph.

Original: Most of the experiments were performed using NVIDIA Tesla T4 Tensor Core GPUs with 16 GB of
Revised: Most of the experiments were performed using NVIDIA Tesla T4 Tensor Core GPUs with 16 GB of

Original: GDDR6 memory, NVIDIA P100 GPUs with 16 GB of CoWoS HBM2 memory, or NVIDIA Tesla
Revised: GDDR6 memory, NVIDIA P100 GPUs with 16 GB of CoWoS HBM2 memory, or NVIDIA Tesla

Original: K80 GPUs with 24 GB of GDDR5 memory.
Revised: K80 GPUs with 24 GB of GDDR5 memory.

Original: All these GPUs have limited memories that are easily exceeded during backpropagation for datasets other than Cora and CiteSeer (Yang et al.
Revised: All these GPUs have limited memories that are easily exceeded during backpropagation for datasets other than Cora and CiteSeer (Yang et al.

Original: ; Lu & Getoor (2003); Sen et al.
Revised: ; Lu & Getoor (2003); Sen et al.

Original: (2008)), which have 2,708 nodes and 3,327 nodes, respectively.
Revised: (2008)), which have 2,708 nodes and 3,327 nodes, respectively.

Original: For example, using the standard PyTorch Geometric implementation we are not able to backpropagate for the PubMed dataset which has 18,717 nodes.
Revised: For example, using the standard PyTorch Geometric implementation we are not able to backpropagate for the PubMed dataset which has 18,717 nodes.

Original: C.3. S YMBOLIC H ANDLING OF D ISTANCE M ETRICS
Revised: C.3. S YMBOLIC H ANDLING OF D ISTANCE M ETRICS

Original: To avoid memory overflows we resort to Kernel Operations (KeOps) (Charlier et al.
Revised: To avoid memory overflows we resort to Kernel Operations (KeOps) (Charlier et al.

Original: (2021)), which makes it possible to compute reductions of large arrays whose entries are given by a mathematical formula.
Revised: (2021)), which makes it possible to compute reductions of large arrays whose entries are given by a mathematical formula.

Original: We can classify matrices in three categories: dense, sparse, and symbolic.
Revised: We can classify matrices in three categories: dense, sparse, and symbolic.

Original: Dense matrices are dense numerical arrays M ij “ M r i, j s which put a heavy load on computer memory.
Revised: Dense matrices are dense numerical arrays M ij “ M r i, j s which put a heavy load on computer memory.

Original: When they increase in size, they can struggle to fit into RAM or GPU memory.
Revised: When they increase in size, they can struggle to fit into RAM or GPU memory.

Original: This is what happens in our case, when calculating distances between points.
Revised: This is what happens in our case, when calculating distances between points.

Original: Sparse matrices are typically used to try to address this problem.
Revised: Sparse matrices are typically used to try to address this problem.

Original: Sparse matrices use lists of indices p i n , j n q and associated values
Revised: Sparse matrices use lists of indices p i n , j n q and associated values

Original: M n .
Revised: M n .

Original: Hence, in sparse matrices we only store the values for non-zero entries.
Revised: Hence, in sparse matrices we only store the values for non-zero entries.

Original: The main limitation of this approach is that the computing speedup obtained by using sparse matrices is highly dependent on sparsity.
Revised: The main limitation of this approach is that the computing speedup obtained by using sparse matrices is highly dependent on sparsity.

Original: To obtain significant improvements in performance using sparse encoding, the original matrix should effectively be more than 99% empty (Charlier et al.
Revised: To obtain significant improvements in performance using sparse encoding, the original matrix should effectively be more than 99% empty (Charlier et al.

Original: This significantly constrains the applicability of sparse encoding.
Revised: This significantly constrains the applicability of sparse encoding.

Original: Alternatively, KeOps uses symbolic matrices, to represent matrices which can be summarized using an underlying common mathematical structure.
Revised: Alternatively, KeOps uses symbolic matrices, to represent matrices which can be summarized using an underlying common mathematical structure.

Original: In this setup the matrix entries M ij are represented as a function of vectors x i and x j , so that
Revised: In this setup the matrix entries M ij are represented as a function of vectors x i and x j , so that

Original: [Equation]
Revised: [Equation]

Original: Even if these objects are not necessarily sparse, they can be represented using only small data arrays x i and x j , which can result into large improvements in computational efficiency, avoiding memory overflow.
Revised: Even if these objects are not necessarily sparse, they can be represented using only small data arrays x i and x j , which can result into large improvements in computational efficiency, avoiding memory overflow.

Original: In our case, the matrices we are working with are fully populated and using sparse tensors would not enhance performance.
Revised: In our case, the matrices we are working with are fully populated and using sparse tensors would not enhance performance.

Original: On the other hand, the symbolic approach implemented using
Revised: On the other hand, the symbolic approach implemented using

Original: KeOps enables us to represent the original dense matrix containing all the distances between all the points in a much more compact way, and to work with larger graphs than Cora and Citeseer (Yang et al.
Revised: KeOps enables us to represent the original dense matrix containing all the distances between all the points in a much more compact way, and to work with larger graphs than Cora and Citeseer (Yang et al.

Original: ; Lu & Getoor (2003); Sen et al.
Revised: ; Lu & Getoor (2003); Sen et al.

Original: (2008)), namely, PubMed, CS, and Physics (Shchur et al.
Revised: (2008)), namely, PubMed, CS, and Physics (Shchur et al.

Original: Finally, note that in our case the function used by the symbolic matrix is the distance metric appropiate for whichever manifold we are using to construct the latent graph.
Revised: Finally, note that in our case the function used by the symbolic matrix is the distance metric appropiate for whichever manifold we are using to construct the latent graph.

Original: [Equation]
Revised: [Equation]

Original: C.3. Q UANTITATIVE S TUDY OF R UNTIME S PEEDUP
Revised: C.3. Q UANTITATIVE S TUDY OF R UNTIME S PEEDUP

Original: We quantify the runtime speedup obtained by using symbolic handling of the distance metrics when generating latent graphs for Cora and CiteSeer.
Revised: We quantify the runtime speedup obtained by using symbolic handling of the distance metrics when generating latent graphs for Cora and CiteSeer.

Original: Note, however, that improved execution time is not the only benefit of symbolic handling.
Revised: Note, however, that improved execution time is not the only benefit of symbolic handling.

Original: As discussed in Section C.3.1, symbolic matrices also help avoid memory overflow for larger graphs.
Revised: As discussed in Section C.3.1, symbolic matrices also help avoid memory overflow for larger graphs.

Original: Indeed, this is the reason we choose Cora and CiteSeer to conduct these experiments: using other datasets we experience GPU memory overflow when using dense matrices, and hence we cannot compare dense to symbolic matrix performance.
Revised: Indeed, this is the reason we choose Cora and CiteSeer to conduct these experiments: using other datasets we experience GPU memory overflow when using dense matrices, and hence we cannot compare dense to symbolic matrix performance.

Original: It should also be highlighted that using both dense and symbolic matrices gives the same results in terms of accuracy since they are equivalent mathematically, the difference lies in the computational efficiency of each method.
Revised: It should also be highlighted that using both dense and symbolic matrices gives the same results in terms of accuracy since they are equivalent mathematically, the difference lies in the computational efficiency of each method.

Original: We display the results in Table 5 and Table 6, in which we record execution times for a 100 epochs using a NVIDIA P100 GPU.
Revised: We display the results in Table 5 and Table 6, in which we record execution times for a 100 epochs using a NVIDIA P100 GPU.

Original: We evaluate the effect of increasing the number of dDGM layers on the runtime.
Revised: We evaluate the effect of increasing the number of dDGM layers on the runtime.

Original: We also compare the dDGM module using Euclidean (GCN-dDGM-E) and a product manifold of Euclidean, hyperbolic, and spherical space (GCN-dDGM-EHS) to generate the latent graphs.
Revised: We also compare the dDGM module using Euclidean (GCN-dDGM-E) and a product manifold of Euclidean, hyperbolic, and spherical space (GCN-dDGM-EHS) to generate the latent graphs.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We can see that as we add more dDGMs
Revised: We can see that as we add more dDGMs

Original: the difference between using dense and symbolic matrices becomes more substantial.
Revised: the difference between using dense and symbolic matrices becomes more substantial.

Original: Likewise, the benefit from using symbolic matrices becomes more apparent when using product manifolds, this is because more distances must be computed.
Revised: Likewise, the benefit from using symbolic matrices becomes more apparent when using product manifolds, this is because more distances must be computed.

Original: The product manifolds are calculated based on the Cartesian product of three model spaces and hence, to obtain the overall distance between each of the nodes, we must compute geodesics in all constant curvature manifolds independently.
Revised: The product manifolds are calculated based on the Cartesian product of three model spaces and hence, to obtain the overall distance between each of the nodes, we must compute geodesics in all constant curvature manifolds independently.

Original: Another observation that we can make is that the computation time for CiteSeer increases substantially as compared to Cora using dense matrices.
Revised: Another observation that we can make is that the computation time for CiteSeer increases substantially as compared to Cora using dense matrices.

Original: CiteSeer only has 995 more nodes than Cora, yet using 3 dDGM-EHS layers the execution time for 100 epochs increases by 5.53 seconds for dense matrices.
Revised: CiteSeer only has 995 more nodes than Cora, yet using 3 dDGM-EHS layers the execution time for 100 epochs increases by 5.53 seconds for dense matrices.

Original: This clearly shows that using dense matrices can quickly become hard to scale for larger graphs, which can have orders of magnitude more nodes than CiteSeer.
Revised: This clearly shows that using dense matrices can quickly become hard to scale for larger graphs, which can have orders of magnitude more nodes than CiteSeer.

Original: In line with the literature, using symbolic matrices is more computationally tractable for larger graphs Kazi et al.
Revised: In line with the literature, using symbolic matrices is more computationally tractable for larger graphs Kazi et al.

Original: Also, we run some additional experiments to quantify the increase in computation time as a function of k , that is, the number of edges per latent graph node when applying the Gumbel Top-k trick.
Revised: Also, we run some additional experiments to quantify the increase in computation time as a function of k , that is, the number of edges per latent graph node when applying the Gumbel Top-k trick.

Original: As we can see in Table 7 and Table 8, k does not seem to have a statistically significant impact on the execution time.
Revised: As we can see in Table 7 and Table 8, k does not seem to have a statistically significant impact on the execution time.

Original: As before, we find that using symbolic matrices is consistently more efficient.
Revised: As before, we find that using symbolic matrices is consistently more efficient.

Original: C.3. T RAINING ON L ARGE G RAPHS
Revised: C.3. T RAINING ON L ARGE G RAPHS

Original: Although symbolic handling of distance metrics is certainly necessary, for larger graphs such as the graphs for node property prediction of the Open Graph Benchmark (OGB) (Hu et al.
Revised: Although symbolic handling of distance metrics is certainly necessary, for larger graphs such as the graphs for node property prediction of the Open Graph Benchmark (OGB) (Hu et al.

Original: (2020)) which have O p 10 5 q ´ O p 10 8 q nodes
Revised: (2020)) which have O p 10 5 q ´ O p 10 8 q nodes

Original: and O p 10 6 q ´ O p 10 9 q edges, we must combine KeOps with graph subsampling techniques to make backpropagating computationally tractable.
Revised: and O p 10 6 q ´ O p 10 9 q edges, we must combine KeOps with graph subsampling techniques to make backpropagating computationally tractable.

Original: We apply a neighbor sampler to track message passing dependencies for the subsampled nodes.
Revised: We apply a neighbor sampler to track message passing dependencies for the subsampled nodes.

Original: This allows computation to be more lightweight.
Revised: This allows computation to be more lightweight.

Original: Based on the message passing equation
Revised: Based on the message passing equation

Original: [Figure]
Revised: [Figure]

Original: using a Tesla T4 GPU.
Revised: using a Tesla T4 GPU.

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: (35) to calculate x p l ` 1 q i we must aggregate, and hence, have stored the node features of its neighbors when subsampling the graph.
Revised: (35) to calculate x p l ` 1 q i we must aggregate, and hence, have stored the node features of its neighbors when subsampling the graph.

Original: Note that unlike in the original node prediction setup in which we give as input all the nodes and predict properties for all nodes in the complete graph, here we have a different number of input and output nodes, which gives rise to a bipartite structure for multi-layer minibatch message passing.
Revised: Note that unlike in the original node prediction setup in which we give as input all the nodes and predict properties for all nodes in the complete graph, here we have a different number of input and output nodes, which gives rise to a bipartite structure for multi-layer minibatch message passing.

Original: Such a bipartite graph, which samples only the necessary input and output nodes from the original graph, is called a message flow graph (Ladkin & Leue (2005)).
Revised: Such a bipartite graph, which samples only the necessary input and output nodes from the original graph, is called a message flow graph (Ladkin & Leue (2005)).

Original: For every node that we compute in a given batch we will need to track its message flow graph alongside all relevant dependencies.
Revised: For every node that we compute in a given batch we will need to track its message flow graph alongside all relevant dependencies.

Original: D T HE A EROTHERMODYNAMICS D ATASET
Revised: D T HE A EROTHERMODYNAMICS D ATASET

Original: We generate a multi-class classification dataset based on Computational Fluid Dynamics (CFD) simulations conducted for the rocket designs developed for the Karman Space Programme.
Revised: We generate a multi-class classification dataset based on Computational Fluid Dynamics (CFD) simulations conducted for the rocket designs developed for the Karman Space Programme.

Original: Specifically, the dataset is generated based on the shock wave velocity distribution around the nose of a rocket at 10 degrees of angle of attack.
Revised: Specifically, the dataset is generated based on the shock wave velocity distribution around the nose of a rocket at 10 degrees of angle of attack.

Original: The simulation meshgrid has varying degrees of resolution, and the shock wave is not symmetric due to the angle of attack.
Revised: The simulation meshgrid has varying degrees of resolution, and the shock wave is not symmetric due to the angle of attack.

Original: Although CFD software uses a meshgrid to discretize space and run the aerothermodynamics simulations, it can be challenging to extract the connectivity of the original graph, since most often the software is designed to only output a pointcloud.
Revised: Although CFD software uses a meshgrid to discretize space and run the aerothermodynamics simulations, it can be challenging to extract the connectivity of the original graph, since most often the software is designed to only output a pointcloud.

Original: Moreover, given that across a shock wave, the static pressure, temperature, and gas density increases almost instantaneously and there is an abrupt decrease in the flow area, the original graph can present high heterophily.
Revised: Moreover, given that across a shock wave, the static pressure, temperature, and gas density increases almost instantaneously and there is an abrupt decrease in the flow area, the original graph can present high heterophily.

Original: Hence, latent graph inference can be beneficial.
Revised: Hence, latent graph inference can be beneficial.

Original: For the dataset, we only consider the shock region at the leading edge of the rocket.
Revised: For the dataset, we only consider the shock region at the leading edge of the rocket.

Original: To obtain the class labels, we separate the shock flow absolute velocity into 4 regions: ď 300 m/s, 300 ´ 450 m/s, 450 ´ 650 m/s and ą 650 m/s. The original simulation has 207,745 datapoints, but we only focus on the shock around the nose of the rocket and apply graph coarsening, see Figure 4.
Revised: To obtain the class labels, we separate the shock flow absolute velocity into 4 regions: ď 300 m/s, 300 ´ 450 m/s, 450 ´ 650 m/s and ą 650 m/s. The original simulation has 207,745 datapoints, but we only focus on the shock around the nose of the rocket and apply graph coarsening, see Figure 4.

Original: The network input is a pointcloud with pressure values.
Revised: The network input is a pointcloud with pressure values.

Original: Note that the coordinates with respect to the rocket are not given to the network.
Revised: Note that the coordinates with respect to the rocket are not given to the network.

Original: The model is tasked with classifying the shock into four intensity regions which are based on the absolute velocity distribution.
Revised: The model is tasked with classifying the shock into four intensity regions which are based on the absolute velocity distribution.

Original: Table 9 summarizes the dataset.
Revised: Table 9 summarizes the dataset.

Original: [Figure]
Revised: [Figure]

Original: E XPERIMENTS AND R ESULTS
Revised: E XPERIMENTS AND R ESULTS

Original: In this appendix we include additional experiments for the latent graph inference system.
Revised: In this appendix we include additional experiments for the latent graph inference system.

Original: In Section E.1 we explore using more than one latent graph inference system per GNN model.
Revised: In Section E.1 we explore using more than one latent graph inference system per GNN model.

Original: In Section E.2 we include additional results for the homophilic graph datasets in which we vary the value of k for the graph generation algorithm.
Revised: In Section E.2 we include additional results for the homophilic graph datasets in which we vary the value of k for the graph generation algorithm.

Original: In Section E.3 we experiment with using a greater number of model spaces to generate product manifolds for the embedding space.
Revised: In Section E.3 we experiment with using a greater number of model spaces to generate product manifolds for the embedding space.

Original: Section E.4 includes results for heterophilic datasets, Section E.5 for OGB, and Section E.6 for inductive learning.
Revised: Section E.4 includes results for heterophilic datasets, Section E.5 for OGB, and Section E.6 for inductive learning.

Original: The First ,
Revised: The First ,

Original: Second and Third best models for each dataset are highlighted in each table.
Revised: Second and Third best models for each dataset are highlighted in each table.

Original: E. N UMBER OF D DGM M ODULES
Revised: E. N UMBER OF D DGM M ODULES

Original: In these experiments
Revised: In these experiments

Original: [Table]
Revised: [Table]

Original: we investigate whether there is any benefit in stacking multiple dDGM modules.
Revised: we investigate whether there is any benefit in stacking multiple dDGM modules.

Original: Using multiple dDGMs effectively means that within the model
Revised: Using multiple dDGMs effectively means that within the model

Original: [Table]
Revised: [Table]

Original: the network layers would be learning based on different latent graphs.
Revised: the network layers would be learning based on different latent graphs.

Original: In principle, according to the results in Table 10 and
Revised: In principle, according to the results in Table 10 and

Original: [Table]
Revised: [Table]

Original: there is no clear improvement in performance for Cora and CiteSeer.
Revised: there is no clear improvement in performance for Cora and CiteSeer.

Original: In fact, the accuracy of the models can decrease.
Revised: In fact, the accuracy of the models can decrease.

Original: It is only for PubMed that there is sometimes some improvement in the accuracy.
Revised: It is only for PubMed that there is sometimes some improvement in the accuracy.

Original: These results align with previous studies by Kazi et al.
Revised: These results align with previous studies by Kazi et al.

Original: Finally, we also run a few experiments for Physics and CS.
Revised: Finally, we also run a few experiments for Physics and CS.

Original: Again, we find no substantial improvement using an additional dDGM module, see Table 12.
Revised: Again, we find no substantial improvement using an additional dDGM module, see Table 12.

Original: Given our findings, we use a single dDGM module, since it is more computationally efficient.
Revised: Given our findings, we use a single dDGM module, since it is more computationally efficient.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: E. H OMOPHILIC G RAPH D ATASETS E XTENDED R ESULTS
Revised: E. H OMOPHILIC G RAPH D ATASETS E XTENDED R ESULTS

Original: In this section we include extended results for the Cora, CiteSeer, PubMed, Physics
Revised: In this section we include extended results for the Cora, CiteSeer, PubMed, Physics

Original: [Table]
Revised: [Table]

Original: and CS datasets.
Revised: and CS datasets.

Original: Table 13 presents results using single model spaces for the embedding space, and Table 14 and
Revised: Table 13 presents results using single model spaces for the embedding space, and Table 14 and

Original: Table 15 using product manifolds.
Revised: Table 15 using product manifolds.

Original: As discussed in the main text, the dDGM ˚ does not use the original dataset graph as inductive bias, since it is only provided with a pointcloud.
Revised: As discussed in the main text, the dDGM ˚ does not use the original dataset graph as inductive bias, since it is only provided with a pointcloud.

Original: On the other hand, the dDGM does use the original graph.
Revised: On the other hand, the dDGM does use the original graph.

Original: Different k values are applied.
Revised: Different k values are applied.

Original: [Table]
Revised: [Table]

Original: E. M ORE C OMPLEX P RODUCT M ANIFOLDS R ESULTS
Revised: E. M ORE C OMPLEX P RODUCT M ANIFOLDS R ESULTS

Original: The main objective of this section is trying to test the limits of our approach.
Revised: The main objective of this section is trying to test the limits of our approach.

Original: In Table 16 we display the results multiplying up to five model spaces for the CS dataset.
Revised: In Table 16 we display the results multiplying up to five model spaces for the CS dataset.

Original: From the results, we can see that adding more product manifolds can result in improved performance.
Revised: From the results, we can see that adding more product manifolds can result in improved performance.

Original: The GCN-dDGM-EHHSS network with k “ 5 obtains an accuracy of 93 .
Revised: The GCN-dDGM-EHHSS network with k “ 5 obtains an accuracy of 93 .

Original: 10 ˘ 0 .
Revised: 10 ˘ 0 .

Original: 74% , as compared to the best single model space based model GCN-dDGM-E with k “ 5 which achieves a result of only 87 .
Revised: 74% , as compared to the best single model space based model GCN-dDGM-E with k “ 5 which achieves a result of only 87 .

Original: 88 ˘ 2 .
Revised: 88 ˘ 2 .

Original: E. H ETEROPHILIC G RAPH D ATASETS E XTENDED R ESULTS
Revised: E. H ETEROPHILIC G RAPH D ATASETS E XTENDED R ESULTS

Original: In Table 17 we display the results using both the dDGM and the dDGM ˚ module for heterophilic datasets.
Revised: In Table 17 we display the results using both the dDGM and the dDGM ˚ module for heterophilic datasets.

Original: The dDGM module uses the original dataset graph as inductive bias for the generation of the latent graphs.
Revised: The dDGM module uses the original dataset graph as inductive bias for the generation of the latent graphs.

Original: As expected, this leads to worse results than those using the dDGM ˚ for heterophilic graphs, since the original graph is not good for diffusion using GCNs.
Revised: As expected, this leads to worse results than those using the dDGM ˚ for heterophilic graphs, since the original graph is not good for diffusion using GCNs.

Original: It is better to start directly from a pointcloud and completely ignore the original adjacency matrix A p 0 q since it does not provide the model with a good inductive bias.
Revised: It is better to start directly from a pointcloud and completely ignore the original adjacency matrix A p 0 q since it does not provide the model with a good inductive bias.

Original: E. R ESULTS FOR L ARGE G RAPHS FROM THE O PEN G RAPH B ENCHMARK
Revised: E. R ESULTS FOR L ARGE G RAPHS FROM THE O PEN G RAPH B ENCHMARK

Original: Table 18 and Table 19 display results for the OGB-Arxiv and OGB-Products datasets.
Revised: Table 18 and Table 19 display results for the OGB-Arxiv and OGB-Products datasets.

Original: Although the
Revised: Although the

Original: OGB-Products dataset is considerably larger than the OGB-Arxiv both in terms of the number of nodes and edges, our models achieve slightly better performance.
Revised: OGB-Products dataset is considerably larger than the OGB-Arxiv both in terms of the number of nodes and edges, our models achieve slightly better performance.

Original: This may be due to the fact that
Revised: This may be due to the fact that

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: the OGB-Products dataset is an undirected graph, whereas the OGB-Arxiv dataset is directed.
Revised: the OGB-Products dataset is an undirected graph, whereas the OGB-Arxiv dataset is directed.

Original: The dDGM module generates undirected edges between nodes by construction (if we neglect the noise in the Gumbel Top-k trick), which may be affecting performance in the case of the OGB-Arxiv dataset.
Revised: The dDGM module generates undirected edges between nodes by construction (if we neglect the noise in the Gumbel Top-k trick), which may be affecting performance in the case of the OGB-Arxiv dataset.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: E. I NDUCTIVE L EARNING : R ESULTS FOR THE QM9 AND A LCHEMY DATASETS
Revised: E. I NDUCTIVE L EARNING : R ESULTS FOR THE QM9 AND A LCHEMY DATASETS

Original: The datasets discussed in the main task were solely concerned with tranductive learning.
Revised: The datasets discussed in the main task were solely concerned with tranductive learning.

Original: For completeness, we show that the latent graph inference system based on product manifolds is also applicable to inductive learning.
Revised: For completeness, we show that the latent graph inference system based on product manifolds is also applicable to inductive learning.

Original: Molecules are naturally represented as graphs, with atoms as nodes and bonds as edges.
Revised: Molecules are naturally represented as graphs, with atoms as nodes and bonds as edges.

Original: Prediction of molecular properties is a popular application of GNNs in chemistry Wieder et al.
Revised: Prediction of molecular properties is a popular application of GNNs in chemistry Wieder et al.

Original: (2020); Stark et al.
Revised: (2020); Stark et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: ; Godwin et al.
Revised: ; Godwin et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: Specifically, we work with the QM9 (Ramakrishnan et al.
Revised: Specifically, we work with the QM9 (Ramakrishnan et al.

Original: Ruddigkeit et al.
Revised: Ruddigkeit et al.

Original: and Alchemy (Morris et al.
Revised: and Alchemy (Morris et al.

Original: datasets which are well known in the Geometric Deep Learning literature.
Revised: datasets which are well known in the Geometric Deep Learning literature.

Original: Table 20 displays the results.
Revised: Table 20 displays the results.

Original: These tasks are substantially different to the ones previously discussed because they involve inductive learning and regression, whereas before all tasks focused on transductive learning and multi-class classification.
Revised: These tasks are substantially different to the ones previously discussed because they involve inductive learning and regression, whereas before all tasks focused on transductive learning and multi-class classification.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: F L ATENT G RAPH L EARNING P LOTS
Revised: F L ATENT G RAPH L EARNING P LOTS

Original: In this appendix we include additional plots for the learned latent graphs for the heterophilic datasets discussed in the main text as well as for the TadPole and the Aerothermodynamics dataset.
Revised: In this appendix we include additional plots for the learned latent graphs for the heterophilic datasets discussed in the main text as well as for the TadPole and the Aerothermodynamics dataset.

Original: F. L EARNED L ATENT G RAPHS FOR H ETEROPHILIC D ATASETS
Revised: F. L EARNED L ATENT G RAPHS FOR H ETEROPHILIC D ATASETS

Original: In Figure 5, 6, 7, and 8, we display the original graphs provided by the heterophilic datasets, and compare them to the latent graphs generated by the dDGM modules.
Revised: In Figure 5, 6, 7, and 8, we display the original graphs provided by the heterophilic datasets, and compare them to the latent graphs generated by the dDGM modules.

Original: From the plots we can clearly see the high homophily levels of the Texas latent graph in Figure 5, for which we obtain four distinct clusters.
Revised: From the plots we can clearly see the high homophily levels of the Texas latent graph in Figure 5, for which we obtain four distinct clusters.

Original: In the case of the bigger datasets in Figure 7, and 8, the algorithm is still able to create clusters but there is mixing between classes.
Revised: In the case of the bigger datasets in Figure 7, and 8, the algorithm is still able to create clusters but there is mixing between classes.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Note that the generated latent graphs are dependent on both the downstream task and the diffusion layers we are using, that is, the GCNs.
Revised: Note that the generated latent graphs are dependent on both the downstream task and the diffusion layers we are using, that is, the GCNs.

Original: Since we have created a fully-differentiable system the models are optimizing the latent graph generation together with the rest of the model parameters.
Revised: Since we have created a fully-differentiable system the models are optimizing the latent graph generation together with the rest of the model parameters.

Original: Hence, if we were to change the downstream task or the diffusion layers we would expect different latent graphs.
Revised: Hence, if we were to change the downstream task or the diffusion layers we would expect different latent graphs.

Original: In Figure 2 from Section 4.1 we showed the latent graph learning evolution plots as a function of training epochs for the Texas dataset.
Revised: In Figure 2 from Section 4.1 we showed the latent graph learning evolution plots as a function of training epochs for the Texas dataset.

Original: We provide additional plots for other datasets.
Revised: We provide additional plots for other datasets.

Original: This shows that the latent graph inference system is applicable across a wide range of datasets and that is learns to organize the connectivity of the latent graph during training.
Revised: This shows that the latent graph inference system is applicable across a wide range of datasets and that is learns to organize the connectivity of the latent graph during training.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Lastly, we display the learned latent graphs using original heterophilic graph as inductive bias for both the Texas and Wisconsin datasets.
Revised: Lastly, we display the learned latent graphs using original heterophilic graph as inductive bias for both the Texas and Wisconsin datasets.

Original: In Figure 12 and Figure 13, we compare the final inferred latent graphs for the datasets when the original heterophilic dataset graph is used as inductive bias, against starting from a pointcloud.
Revised: In Figure 12 and Figure 13, we compare the final inferred latent graphs for the datasets when the original heterophilic dataset graph is used as inductive bias, against starting from a pointcloud.

Original: Models that use the dDGM module, which makes use of the original graph, are not able to achieve high homophily levels as compared to those using the dDGM ˚ module and ignoring the original graph.
Revised: Models that use the dDGM module, which makes use of the original graph, are not able to achieve high homophily levels as compared to those using the dDGM ˚ module and ignoring the original graph.

Original: This explains the difference in performance in Tablefrom Appendix E.4.
Revised: This explains the difference in performance in Tablefrom Appendix E.4.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: L EARNED L ATENT G RAPHS FOR R EAL -W ORLD DATASETS
Revised: L EARNED L ATENT G RAPHS FOR R EAL -W ORLD DATASETS

Original: Next we displayed the learned latent graph for the TadPole and the Aerothermodynamics datasets, see Figure 14, Figure 15, Figure and Figure 17.
Revised: Next we displayed the learned latent graph for the TadPole and the Aerothermodynamics datasets, see Figure 14, Figure 15, Figure and Figure 17.

Original: G M ODEL A RCHITECTURES
Revised: G M ODEL A RCHITECTURES

Original: For reproducibility, in this appendix we include a summary of all the models used to obtain the results discussed in this paper.
Revised: For reproducibility, in this appendix we include a summary of all the models used to obtain the results discussed in this paper.

Original: This includes the GNN network archictectures as well as the internal structure of the dDGM modules.
Revised: This includes the GNN network archictectures as well as the internal structure of the dDGM modules.

Original: G. N ETWORK A RCHITECTURES
Revised: G. N ETWORK A RCHITECTURES

Original: In this section we provide the neural network architectures used for the experimental validation as well as other training specifications.
Revised: In this section we provide the neural network architectures used for the experimental validation as well as other training specifications.

Original: G.1. N ETWORKS FOR C LASSICAL G RAPH D ATASETS
Revised: G.1. N ETWORKS FOR C LASSICAL G RAPH D ATASETS

Original: All neural network models used for homophilic graph datasets, MLP, GCN, and GCN-dDGMs, follow the architecture depicted in Table 21.
Revised: All neural network models used for homophilic graph datasets, MLP, GCN, and GCN-dDGMs, follow the architecture depicted in Table 21.

Original: We apply a learning rate of lr “ 10 ´ 2 and a weight decay of wd “ 10 ´ 4 .
Revised: We apply a learning rate of lr “ 10 ´ 2 and a weight decay of wd “ 10 ´ 4 .

Original: Models are trained for about 1,500 epochs.
Revised: Models are trained for about 1,500 epochs.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: G.1. N ETWORKS FOR H ETEROPHILIC G RAPH D ATASETS
Revised: G.1. N ETWORKS FOR H ETEROPHILIC G RAPH D ATASETS

Original: The networks used for heterophilic datasets follow the architecture summarized in Table 22.
Revised: The networks used for heterophilic datasets follow the architecture summarized in Table 22.

Original: We apply lr “ 10 ´ 2 and wd “ 10 ´ 3 , and we train the models for about 1,000 epochs.
Revised: We apply lr “ 10 ´ 2 and wd “ 10 ´ 3 , and we train the models for about 1,000 epochs.

Original: [Table]
Revised: [Table]

Original: G.1. N ETWORKS FOR OGB-A RXIV
Revised: G.1. N ETWORKS FOR OGB-A RXIV

Original: Table 23 shows the networks used for the OGB-Arxiv dataset.
Revised: Table 23 shows the networks used for the OGB-Arxiv dataset.

Original: We use lr “ 10 ´ 3 , wd “
Revised: We use lr “ 10 ´ 3 , wd “

Original: [Table]
Revised: [Table]

Original: and train for 100 epochs.
Revised: and train for 100 epochs.

Original: For graph subsampling, we sample up to 1,000 neighbours per node and use a batch size of 1,000.
Revised: For graph subsampling, we sample up to 1,000 neighbours per node and use a batch size of 1,000.

Original: dataset using GATv diffusion layers.
Revised: dataset using GATv diffusion layers.

Original: As specified in the table, k “ 20 is used for all experiments.
Revised: As specified in the table, k “ 20 is used for all experiments.

Original: [Table]
Revised: [Table]

Original: G.1. N ETWORKS FOR OGB-P RODUCTS
Revised: G.1. N ETWORKS FOR OGB-P RODUCTS

Original: Table 24 shows the networks used for the OGB-Products dataset.
Revised: Table 24 shows the networks used for the OGB-Products dataset.

Original: We use lr “ 10 ´ 2 , wd “ 0 , and train for 30 epochs.
Revised: We use lr “ 10 ´ 2 , wd “ 0 , and train for 30 epochs.

Original: For graph subsampling, we sample up to 200 neighbors per node and use a batch size of 1,000.
Revised: For graph subsampling, we sample up to 200 neighbors per node and use a batch size of 1,000.

Original: [Table]
Revised: [Table]

Original: G.1. N ETWORKS FOR I NDUCTIVE L EARNING
Revised: G.1. N ETWORKS FOR I NDUCTIVE L EARNING

Original: Table 25 summarizes the networks used for the QM9 and Alchemy datasets.
Revised: Table 25 summarizes the networks used for the QM9 and Alchemy datasets.

Original: We use lr “ 5 ˆ 10 ´ 2 , wd “ 10 ´ 3 , and train for 30 to 60 epochs.
Revised: We use lr “ 5 ˆ 10 ´ 2 , wd “ 10 ´ 3 , and train for 30 to 60 epochs.

Original: [Table]
Revised: [Table]

Original: G.1. N ETWORKS FOR B RAIN I MAGING
Revised: G.1. N ETWORKS FOR B RAIN I MAGING

Original: Table 26 summarizes the networks used for the TadPole dataset.
Revised: Table 26 summarizes the networks used for the TadPole dataset.

Original: We use lr “ 10 ´ 3 and wd “ 2 ˆ 10 ´ 4 , and train for about 800 epochs.
Revised: We use lr “ 10 ´ 3 and wd “ 2 ˆ 10 ´ 4 , and train for about 800 epochs.

Original: for TadPole.
Revised: for TadPole.

Original: [Table]
Revised: [Table]

Original: G.1. N ETWORKS FOR A EROSPACE E NGINEERING
Revised: G.1. N ETWORKS FOR A EROSPACE E NGINEERING

Original: Table 27 summarizes the networks used for the Aerothermodynamics dataset.
Revised: Table 27 summarizes the networks used for the Aerothermodynamics dataset.

Original: We use lr “ 10 ´and wd “ 0 , and train for about 1,000 epochs
Revised: We use lr “ 10 ´and wd “ 0 , and train for about 1,000 epochs

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: G. D DGM A RCHITECTURES
Revised: G. D DGM A RCHITECTURES

Original: The dDGM and dDGM ˚ internal parameterized mapping functions are slightly modified for different datasets.
Revised: The dDGM and dDGM ˚ internal parameterized mapping functions are slightly modified for different datasets.

Original: All architectures are recorded in Table 28, 29, 30, 31, 32,
Revised: All architectures are recorded in Table 28, 29, 30, 31, 32,

Original: [Table]
Revised: [Table]

Original: and 34.
Revised: and 34.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]