Original: C OPULA GNN: T OWARDS I NTEGRATING R EPRESENTATIONAL AND C ORRELATIONAL R OLES OF G RAPHS IN G RAPH N EURAL N ETWORKS
Revised: C OPULA GNN: T OWARDS I NTEGRATING R EPRESENTATIONAL AND C ORRELATIONAL R OLES OF G RAPHS IN G RAPH N EURAL N ETWORKS

Original: qmei@ edu
Revised: qmei@ edu

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph-structured data are ubiquitous.
Revised: Graph-structured data are ubiquitous.

Original: However, graphs encode diverse types of information and thus play different roles in data representation.
Revised: However, graphs encode diverse types of information and thus play different roles in data representation.

Original: In this paper, we distinguish the representational and the correlational roles played by the graphs in node-level prediction tasks, and we investigate how Graph Neural Network (GNN) models can effectively leverage both types of information.
Revised: In this paper, we distinguish the representational and the correlational roles played by the graphs in node-level prediction tasks, and we investigate how Graph Neural Network (GNN) models can effectively leverage both types of information.

Original: Conceptually, the representational information provides guidance for the model to construct better node features; while the correlational information indicates the correlation between node outcomes conditional on node features.
Revised: Conceptually, the representational information provides guidance for the model to construct better node features; while the correlational information indicates the correlation between node outcomes conditional on node features.

Original: Through a simulation study, we ﬁnd that many popular GNN models are incapable of effectively utilizing the correlational information.
Revised: Through a simulation study, we ﬁnd that many popular GNN models are incapable of effectively utilizing the correlational information.

Original: By leveraging the idea of the copula, a principled way to describe the dependence among multivariate random variables, we offer a general solution.
Revised: By leveraging the idea of the copula, a principled way to describe the dependence among multivariate random variables, we offer a general solution.

Original: The proposed Copula Graph Neural Network (CopulaGNN) can take a wide range of GNN models as base models and utilize both representational and correlational information stored in the graphs.
Revised: The proposed Copula Graph Neural Network (CopulaGNN) can take a wide range of GNN models as base models and utilize both representational and correlational information stored in the graphs.

Original: Experimental results on two types of regression tasks verify the effectiveness of the proposed method 1 .
Revised: Experimental results on two types of regression tasks verify the effectiveness of the proposed method 1 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graphs, as ﬂexible data representations that store rich relational information, have been commonly used in data science tasks.
Revised: Graphs, as ﬂexible data representations that store rich relational information, have been commonly used in data science tasks.

Original: Machine learning methods on graphs (Chami et al., 2020), especially
Revised: Machine learning methods on graphs (Chami et al., 2020), especially

Original: Graph Neural Networks (GNNs), have attracted increasing interest in the research community.
Revised: Graph Neural Networks (GNNs), have attracted increasing interest in the research community.

Original: They are widely applied to real-world problems such as recommender systems (Ying et al., 2018), social network analysis (Li et al., 2017), and transportation forecasting (Yu et al., 2017).
Revised: They are widely applied to real-world problems such as recommender systems (Ying et al., 2018), social network analysis (Li et al., 2017), and transportation forecasting (Yu et al., 2017).

Original: Among the heterogeneous types of graph-structured data, it is worth noting that graphs usually play diverse roles in different contexts, different datasets, and different tasks.
Revised: Among the heterogeneous types of graph-structured data, it is worth noting that graphs usually play diverse roles in different contexts, different datasets, and different tasks.

Original: Some of the roles are relational, as a graph may indicate certain statistical relationships of connected observations; some are representational, as the topological structure of a graph may encode important features/patterns of the data; some are even causal, as a graph may reﬂect causal relationships speciﬁed by domain experts.
Revised: Some of the roles are relational, as a graph may indicate certain statistical relationships of connected observations; some are representational, as the topological structure of a graph may encode important features/patterns of the data; some are even causal, as a graph may reﬂect causal relationships speciﬁed by domain experts.

Original: It is crucial to recognize the distinct roles of a graph in order to correctly utilize the signals in the graph-structured data.
Revised: It is crucial to recognize the distinct roles of a graph in order to correctly utilize the signals in the graph-structured data.

Original: In this paper, we distinguish the representational role and the correlational role of graphs in the context of node-level (semi-)supervised learning, and we investigate how to design better GNNs that take advantage of both roles.
Revised: In this paper, we distinguish the representational role and the correlational role of graphs in the context of node-level (semi-)supervised learning, and we investigate how to design better GNNs that take advantage of both roles.

Original: In a node-level prediction task, the observed graph in the data may relate to the outcomes of interest (e.g., node labels) in multiple ways.
Revised: In a node-level prediction task, the observed graph in the data may relate to the outcomes of interest (e.g., node labels) in multiple ways.

Original: Conceptually, we call that the graph plays a representational role if one can leverage it to construct better feature representations.
Revised: Conceptually, we call that the graph plays a representational role if one can leverage it to construct better feature representations.

Original: For example, in social network analysis, aggregating user features from one’s friends is usually helpful (thanks to the well-known homophily phenomenon (McPherson et al., 2001)).
Revised: For example, in social network analysis, aggregating user features from one’s friends is usually helpful (thanks to the well-known homophily phenomenon (McPherson et al., 2001)).

Original: In addition, the structural properties of a user’s local network, e.g., structural diversity (Ugander et al., 2012) and structural holes (Burt, 2009; Lou & Tang, 2013), often provide useful information for making predictions about certain outcomes of that user.
Revised: In addition, the structural properties of a user’s local network, e.g., structural diversity (Ugander et al., 2012) and structural holes (Burt, 2009; Lou & Tang, 2013), often provide useful information for making predictions about certain outcomes of that user.

Original: On the other hand, sometimes a graph directly encodes correlations between the outcomes of connected nodes, and we call it playing a correlational role .
Revised: On the other hand, sometimes a graph directly encodes correlations between the outcomes of connected nodes, and we call it playing a correlational role .

Original: For example, hyper-linked Webpages are likely to be visited together even if they have dissimilar content.
Revised: For example, hyper-linked Webpages are likely to be visited together even if they have dissimilar content.

Original: In spatiotemporal predictions, the outcome of nearby locations, conditional on all the features, may still be correlated.
Revised: In spatiotemporal predictions, the outcome of nearby locations, conditional on all the features, may still be correlated.

Original: We note that the graph structure may provide useful predictive information through both roles but in distinct ways.
Revised: We note that the graph structure may provide useful predictive information through both roles but in distinct ways.

Original: While both the representational and the correlational roles are common in graph-structured data, we ﬁnd that, through a simulation study, many existing GNN models are incapable of utilizing the correlational information encoded in a graph.
Revised: While both the representational and the correlational roles are common in graph-structured data, we ﬁnd that, through a simulation study, many existing GNN models are incapable of utilizing the correlational information encoded in a graph.

Original: Speciﬁcally, we design a synthetic dataset for the node-level regression.
Revised: Speciﬁcally, we design a synthetic dataset for the node-level regression.

Original: The node-level outcomes are drawn from a multivariate normal distribution, with the mean and the covariance as functions of the graph to reﬂect the representational and correlation roles respectively.
Revised: The node-level outcomes are drawn from a multivariate normal distribution, with the mean and the covariance as functions of the graph to reﬂect the representational and correlation roles respectively.

Original: We ﬁnd that when the graph only provides correlational information of the node outcomes, many popular GNN models underperform a multi-layer perceptron which does not consider the graph at all.
Revised: We ﬁnd that when the graph only provides correlational information of the node outcomes, many popular GNN models underperform a multi-layer perceptron which does not consider the graph at all.

Original: To mitigate this deﬁciency of GNNs, we propose a principled solution, the Copula Graph Neural
Revised: To mitigate this deﬁciency of GNNs, we propose a principled solution, the Copula Graph Neural

Original: Network (CopulaGNN), which can take a wide range of GNNs as the base model and improve their capabilities of modeling the correlational graph information.
Revised: Network (CopulaGNN), which can take a wide range of GNNs as the base model and improve their capabilities of modeling the correlational graph information.

Original: The key insight of the proposed method is that, by decomposing the joint distribution of node outcomes into the product of marginal densities and a copula density, the representational information and correlational information can be separately modeled.
Revised: The key insight of the proposed method is that, by decomposing the joint distribution of node outcomes into the product of marginal densities and a copula density, the representational information and correlational information can be separately modeled.

Original: The former is modeled by the marginal densities through a base GNN while the latter is modeled by a Gaussian copula.
Revised: The former is modeled by the marginal densities through a base GNN while the latter is modeled by a Gaussian copula.

Original: The proposed method also enjoys the beneﬁt of easy extension to various types of node outcome variables including continuous variables, discrete count variables, or even mixed-type variables.
Revised: The proposed method also enjoys the beneﬁt of easy extension to various types of node outcome variables including continuous variables, discrete count variables, or even mixed-type variables.

Original: We instantiate CopulaGNN with normal and Poisson marginal distributions for continuous and count regression tasks respectively.
Revised: We instantiate CopulaGNN with normal and Poisson marginal distributions for continuous and count regression tasks respectively.

Original: We also implement two types of copula parameterizations combined with two types of base GNNs.
Revised: We also implement two types of copula parameterizations combined with two types of base GNNs.

Original: We evaluate the proposed method on both synthetic and real-world data with both continuous and count regression tasks.
Revised: We evaluate the proposed method on both synthetic and real-world data with both continuous and count regression tasks.

Original: The experimental results show that CopulaGNNs signiﬁcantly outperform their base GNN counterparts when the graph in the data exhibits both correlational and representational roles.
Revised: The experimental results show that CopulaGNNs signiﬁcantly outperform their base GNN counterparts when the graph in the data exhibits both correlational and representational roles.

Original: We summarize our main contributions as follows:
Revised: We summarize our main contributions as follows:

Original: We raise the question of distinguishing the two roles played by the graph and demonstrate that many existing GNNs are incapable of utilizing the graph information when it plays a pure correlational role.
Revised: We raise the question of distinguishing the two roles played by the graph and demonstrate that many existing GNNs are incapable of utilizing the graph information when it plays a pure correlational role.

Original: We propose a principled solution, the CopulaGNN, to integrate the representational and correlational roles of the graph.
Revised: We propose a principled solution, the CopulaGNN, to integrate the representational and correlational roles of the graph.

Original: We empirically demonstrate the effectiveness of CopulaGNN compared to base GNNs on semi-supervised regression tasks.
Revised: We empirically demonstrate the effectiveness of CopulaGNN compared to base GNNs on semi-supervised regression tasks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There have been extensive existing works that model either the representational role or the correlational role of the graph in node-level (semi-)supervised learning tasks.
Revised: There have been extensive existing works that model either the representational role or the correlational role of the graph in node-level (semi-)supervised learning tasks.

Original: However, there are fewer methods that try to model both sides simultaneously, especially with a GNN.
Revised: However, there are fewer methods that try to model both sides simultaneously, especially with a GNN.

Original: Methods focusing on the representational role.
Revised: Methods focusing on the representational role.

Original: As we mentioned in Section 1, the graph can help construct better node feature representations by both providing extra topological information and guiding node feature aggregation.
Revised: As we mentioned in Section 1, the graph can help construct better node feature representations by both providing extra topological information and guiding node feature aggregation.

Original: There have been vast existing studies on both directions, and among them we can only list a couple of examples.
Revised: There have been vast existing studies on both directions, and among them we can only list a couple of examples.

Original: Various methods have been proposed to leverage the topological information of graph-structured data in machine learning tasks, such as graph kernels (Vishwanathan et al., 2010), node embeddings (Perozzi et al., 2014; Tang et al., 2015;
Revised: Various methods have been proposed to leverage the topological information of graph-structured data in machine learning tasks, such as graph kernels (Vishwanathan et al., 2010), node embeddings (Perozzi et al., 2014; Tang et al., 2015;

Original: Grover & Leskovec, 2016), and GNNs (Xu et al., 2018).
Revised: Grover & Leskovec, 2016), and GNNs (Xu et al., 2018).

Original: Aggregating node features on an attributed graph has also been widely studied, e.g., through feature smoothing (Mei et al., 2008) or GNNs (Kipf
Revised: Aggregating node features on an attributed graph has also been widely studied, e.g., through feature smoothing (Mei et al., 2008) or GNNs (Kipf

Original: & Welling, 2016; Hamilton et al., 2017).
Revised: & Welling, 2016; Hamilton et al., 2017).

Original: In this work, we restrict our focus on the GNN models, which have been the state-of-the-art graph representation learning method on various tasks.
Revised: In this work, we restrict our focus on the GNN models, which have been the state-of-the-art graph representation learning method on various tasks.

Original: Methods focusing on the correlational role.
Revised: Methods focusing on the correlational role.

Original: On the other hand, there has also been extensive literature on modeling the dependence of variables on connected nodes in a graph.
Revised: On the other hand, there has also been extensive literature on modeling the dependence of variables on connected nodes in a graph.

Original: One group of methods is called the graph-based regularization (Zhu et al., 2003; Li et al., 2019), where it is assumed that the variables associated with linked objects change smoothly and pose an explicit similarity regularization among them.
Revised: One group of methods is called the graph-based regularization (Zhu et al., 2003; Li et al., 2019), where it is assumed that the variables associated with linked objects change smoothly and pose an explicit similarity regularization among them.

Original: The correlational role of the graph is also closely related to undirected graphical models (Lauritzen, 1996; Jordan et al., 2004; Wainwright & Jordan, 2008).
Revised: The correlational role of the graph is also closely related to undirected graphical models (Lauritzen, 1996; Jordan et al., 2004; Wainwright & Jordan, 2008).

Original: In graphical models, the edges in a graph provide a representation of the conditional (in)dependence structure among a set of random variables, which are represented by the node set of the graph.
Revised: In graphical models, the edges in a graph provide a representation of the conditional (in)dependence structure among a set of random variables, which are represented by the node set of the graph.

Original: Finally, there has been a line of research that combines graphical models with copulas and leads to more ﬂexible model families (Elidan, 2010; Dobra et al., 2011; Liu et al., 2012; Bauer et al., 2012).
Revised: Finally, there has been a line of research that combines graphical models with copulas and leads to more ﬂexible model families (Elidan, 2010; Dobra et al., 2011; Liu et al., 2012; Bauer et al., 2012).

Original: Our proposed method integrates the beneﬁts of copulas and GNNs to capture both the representational and correlational roles.
Revised: Our proposed method integrates the beneﬁts of copulas and GNNs to capture both the representational and correlational roles.

Original: Methods improving GNNs by leveraging the correlational graph information.
Revised: Methods improving GNNs by leveraging the correlational graph information.

Original: A few methods explicitly leverage the correlational graph information to improve the GNN training, but most of them focus on a classiﬁcation setting (Qu et al., 2019; Ma et al., 2019).
Revised: A few methods explicitly leverage the correlational graph information to improve the GNN training, but most of them focus on a classiﬁcation setting (Qu et al., 2019; Ma et al., 2019).

Original: A recent study (Jia &
Revised: A recent study (Jia &

Original: Benson, 2020) that we have been aware of only lately shares a similar motivation to ours, yet our methodology differs signiﬁcantly.
Revised: Benson, 2020) that we have been aware of only lately shares a similar motivation to ours, yet our methodology differs signiﬁcantly.

Original: In particular, Jia & Benson (2020) apply a multivariate normal distribution to model the correlation of node outcomes, which can be viewed as a special case of our proposed CopulaGNN when a Gaussian copula with normal marginals is used.
Revised: In particular, Jia & Benson (2020) apply a multivariate normal distribution to model the correlation of node outcomes, which can be viewed as a special case of our proposed CopulaGNN when a Gaussian copula with normal marginals is used.

Original: Our method not only generalizes to other marginals (we show the effectiveness of some of them), but also has a more ﬂexible parameterization on the correlation matrix of the copula distribution.
Revised: Our method not only generalizes to other marginals (we show the effectiveness of some of them), but also has a more ﬂexible parameterization on the correlation matrix of the copula distribution.

Original: In addition, we differ with these previous works by explicitly distinguishing the two roles of the graph in the data.
Revised: In addition, we differ with these previous works by explicitly distinguishing the two roles of the graph in the data.

Original:  S IMULATING THE T WO R OLES OF THE G RAPH
Revised:  S IMULATING THE T WO R OLES OF THE G RAPH

Original: In this section, we investigate, through a simulation study, the representational and correlational roles of the graph in the context of node-level semi-supervised learning.
Revised: In this section, we investigate, through a simulation study, the representational and correlational roles of the graph in the context of node-level semi-supervised learning.

Original: 3. N ODE -L EVEL S EMI -S UPERVISED L EARNING
Revised: 3. N ODE -L EVEL S EMI -S UPERVISED L EARNING

Original: We start by formally introducing the problem of node-level semi-supervised learning.
Revised: We start by formally introducing the problem of node-level semi-supervised learning.

Original: A graph is a tuple: G =
Revised: A graph is a tuple: G =

Original: ( V , E ) , where V = { 1 , 2 , . . .
Revised: ( V , E ) , where V = { 1 , 2 , . . .

Original: , n } is the set of n nodes; E ∈ V × V is the set of edges and let s = |E| be the number of edges.
Revised: , n } is the set of n nodes; E ∈ V × V is the set of edges and let s = |E| be the number of edges.

Original: The graph is also associated with X ∈ R n × d and y ∈ R n , which are the node features and outcome labels.
Revised: The graph is also associated with X ∈ R n × d and y ∈ R n , which are the node features and outcome labels.

Original: In the semi-supervised learning setting, we only observe the labels of 0 < m < n nodes.
Revised: In the semi-supervised learning setting, we only observe the labels of 0 < m < n nodes.

Original: Without loss of generality, we assume the labels of nodes { 1 , 2 , . . .
Revised: Without loss of generality, we assume the labels of nodes { 1 , 2 , . . .

Original: , m } are observed and those of { m + 1 , . . .
Revised: , m } are observed and those of { m + 1 , . . .

Original: , n } are missing.
Revised: , n } are missing.

Original: Therefore, the label vector y can be partitioned as y =
Revised: Therefore, the label vector y can be partitioned as y =

Original: ( y T obs , y T miss ) T .
Revised: ( y T obs , y T miss ) T .

Original: The goal of a node-level semi-supervised learning task is to infer y miss based on ( y obs , X , G ) .
Revised: The goal of a node-level semi-supervised learning task is to infer y miss based on ( y obs , X , G ) .

Original: 3. S YNTHETIC D ATA
Revised: 3. S YNTHETIC D ATA

Original: To simulate the representational and correlational roles of the graph, we ﬁrst design a synthetic dataset by specifying the joint distribution of y conditional on X and G .
Revised: To simulate the representational and correlational roles of the graph, we ﬁrst design a synthetic dataset by specifying the joint distribution of y conditional on X and G .

Original: In particular, we let the joint distribution of the node outcomes take the form of y | X , G ∼ N ( µ ( X , G ) , Σ ( G )) , for some µ , Σ to be speciﬁed.
Revised: In particular, we let the joint distribution of the node outcomes take the form of y | X , G ∼ N ( µ ( X , G ) , Σ ( G )) , for some µ , Σ to be speciﬁed.

Original: In this way, the graph G plays a representational role through µ ( X , G ) and a correlational role through Σ ( G ) .
Revised: In this way, the graph G plays a representational role through µ ( X , G ) and a correlational role through Σ ( G ) .

Original: Speciﬁcally, we generate synthetic node-level regression data on a graph with n nodes and s edges (see Appendix A.1 for the whole procedure).
Revised: Speciﬁcally, we generate synthetic node-level regression data on a graph with n nodes and s edges (see Appendix A.1 for the whole procedure).

Original: We ﬁrst randomly generate a feature matrix X ∈ R n × d 0 .
Revised: We ﬁrst randomly generate a feature matrix X ∈ R n × d 0 .

Original: Assume A is the adjacency matrix of the graph, D is the degree matrix, and L = D − A is the graph Laplacian.
Revised: Assume A is the adjacency matrix of the graph, D is the degree matrix, and L = D − A is the graph Laplacian.

Original: Let ˜
Revised: Let ˜

Original: A
Revised: A

Original: =
Revised: =

Original: A + I and ˜ D
Revised: A + I and ˜ D

Original: = D
Revised: = D

Original: + I .
Revised: + I .

Original: Given parameters w y ∈ R d 0 , we generate the node label vector y ∼ N ( µ , Σ ) , where, for some γ > 0 , τ > 0 , and σ 2 > 0 ,
Revised: Given parameters w y ∈ R d 0 , we generate the node label vector y ∼ N ( µ , Σ ) , where, for some γ > 0 , τ > 0 , and σ 2 > 0 ,

Original: (a) µ = ˜ D − 1 ˜
Revised: (a) µ = ˜ D − 1 ˜

Original: A X w y , Σ
Revised: A X w y , Σ

Original: = σ 2 I ; (b) µ = X w y , Σ
Revised: = σ 2 I ; (b) µ = X w y , Σ

Original: = τ ( L + γ I ) − 1 ;
Revised: = τ ( L + γ I ) − 1 ;

Original: [Figure]
Revised: [Figure]

Original: µ
Revised: µ

Original: [Figure]
Revised: [Figure]

Original: ˜
Revised: ˜

Original: [Figure]
Revised: [Figure]

Original: ˜
Revised: ˜

Original: A X w
Revised: A X w

Original: [Figure]
Revised: [Figure]

Original: Σ
Revised: Σ

Original: [Figure]
Revised: [Figure]

Original: σ
Revised: σ

Original: [Figure]
Revised: [Figure]

Original: (b) µ = X w y , Σ
Revised: (b) µ = X w y , Σ

Original: = τ ( L + γ I ) − 1 .
Revised: = τ ( L + γ I ) − 1 .

Original: [Equation]
Revised: [Equation]

Original: Depending on how ( µ , Σ ) are conﬁgured, we get three types of synthetic data settings: (a), (b), and (c).
Revised: Depending on how ( µ , Σ ) are conﬁgured, we get three types of synthetic data settings: (a), (b), and (c).

Original: Intuitively, the graph plays a pure representational role in setting (a) since the label of a node depends on the aggregated features of its local neighborhood and the node labels are independent conditional on the node features.
Revised: Intuitively, the graph plays a pure representational role in setting (a) since the label of a node depends on the aggregated features of its local neighborhood and the node labels are independent conditional on the node features.

Original: In setting (b), the graph plays a pure correlational role; while the means of node labels only depend on their own node features, the node labels are still correlated conditional on the features, and the correlation is determined by the graph structure.
Revised: In setting (b), the graph plays a pure correlational role; while the means of node labels only depend on their own node features, the node labels are still correlated conditional on the features, and the correlation is determined by the graph structure.

Original: Finally, setting (c) is a combination of (a) and (b) where the graph plays both representational and correlational roles.
Revised: Finally, setting (c) is a combination of (a) and (b) where the graph plays both representational and correlational roles.

Original: In the rest of this section, we test the performance of a few widely used GNNs under setting (a) and (b) to examine their capabilities of utilizing the representational and correlational information.
Revised: In the rest of this section, we test the performance of a few widely used GNNs under setting (a) and (b) to examine their capabilities of utilizing the representational and correlational information.

Original: We defer the experimental results under setting (c) to Section 5.2 for ease of reading.
Revised: We defer the experimental results under setting (c) to Section 5.2 for ease of reading.

Original: 3. S IMULATION S TUDY Simulation Setup.
Revised: 3. S IMULATION S TUDY Simulation Setup.

Original: We set the number of nodes n = 300 , the number of edges s = 5000 , and the feature dimension d 0 = 10 .
Revised: We set the number of nodes n = 300 , the number of edges s = 5000 , and the feature dimension d 0 = 10 .

Original: Elements of both W g and w y are generated from i.i.d.
Revised: Elements of both W g and w y are generated from i.i.d.

Original: standard normal distribution.
Revised: standard normal distribution.

Original: For setting (a), we vary σ 2 ∈ { 2 . 5 , 5 , 10 , 20 } .
Revised: For setting (a), we vary σ 2 ∈ { 2 . 5 , 5 , 10 , 20 } .

Original: For settings (b) and (c), we set γ = 0 .
Revised: For settings (b) and (c), we set γ = 0 .

Original: 1 and vary τ ∈ { 0 . 5 , 1 , 2 , 5 } .
Revised: 1 and vary τ ∈ { 0 . 5 , 1 , 2 , 5 } .

Original: We test 4 common GNN models, GCN (Kipf & Welling, 2016), GraphSAGE (Hamilton et al., 2017) (denoted as SAGE), GAT (Veli ˇ ckovi ´
Revised: We test 4 common GNN models, GCN (Kipf & Welling, 2016), GraphSAGE (Hamilton et al., 2017) (denoted as SAGE), GAT (Veli ˇ ckovi ´

Original: c et al., 2018), and
Revised: c et al., 2018), and

Original: APPNP (Klicpera et al., 2018), as well as the multi-layer perceptron (MLP).
Revised: APPNP (Klicpera et al., 2018), as well as the multi-layer perceptron (MLP).

Original: Simulation Results
Revised: Simulation Results

Original: First, we observe that all 4 types of GNNs outperform MLP under setting (a) (Figure 1a), where the graph plays a pure representational role.
Revised: First, we observe that all 4 types of GNNs outperform MLP under setting (a) (Figure 1a), where the graph plays a pure representational role.

Original: This is not surprising as the architectures of the GNNs encode a similar feature aggregation structure as the data.
Revised: This is not surprising as the architectures of the GNNs encode a similar feature aggregation structure as the data.

Original: However, under setting (b) (Figure 1b) where the graph plays a pure correlational role, all 4 types of GNNs underperform MLP.
Revised: However, under setting (b) (Figure 1b) where the graph plays a pure correlational role, all 4 types of GNNs underperform MLP.

Original: This suggests that a majority of popular GNN models might be incapable of fully utilizing the correlational graph information.
Revised: This suggests that a majority of popular GNN models might be incapable of fully utilizing the correlational graph information.

Original: Motivated by our ﬁndings in the simulation study, in the following section, we seek for methods that augment existing GNN models in order to better utilize both representational and correlational information in the graph.
Revised: Motivated by our ﬁndings in the simulation study, in the following section, we seek for methods that augment existing GNN models in order to better utilize both representational and correlational information in the graph.

Original:  C OPULA G RAPH N EURAL N ETWORK
Revised:  C OPULA G RAPH N EURAL N ETWORK

Original: In this section, we propose a principled solution called the Copula Graph Neural Network (CopulaGNN).
Revised: In this section, we propose a principled solution called the Copula Graph Neural Network (CopulaGNN).

Original: At the core of our method is the application of copulas, which are widely used for modeling multivariate dependence.
Revised: At the core of our method is the application of copulas, which are widely used for modeling multivariate dependence.

Original: In the rest of this section, we ﬁrst provide a brief introduction to copulas (more detailed expositions can be found in the monographs by Joe (2014) and Czado (2019)), then present the proposed CopulaGNN and its parameterization, learning, and inference.
Revised: In the rest of this section, we ﬁrst provide a brief introduction to copulas (more detailed expositions can be found in the monographs by Joe (2014) and Czado (2019)), then present the proposed CopulaGNN and its parameterization, learning, and inference.

Original: 4. I NTRODUCTION TO C OPULAS
Revised: 4. I NTRODUCTION TO C OPULAS

Original: In order to decompose the joint distribution of the labels y into representational and correlational components, we make use of copulas , which are widely used in multivariate statistics to model the joint distribution of a random vector.
Revised: In order to decompose the joint distribution of the labels y into representational and correlational components, we make use of copulas , which are widely used in multivariate statistics to model the joint distribution of a random vector.

Original: General formulation.
Revised: General formulation.

Original: Sklar’s theorem (Sklar, 1959) states that any multivariate joint distribution
Revised: Sklar’s theorem (Sklar, 1959) states that any multivariate joint distribution

Original: F of a random vector Y =
Revised: F of a random vector Y =

Original: ( Y 1 , . . .
Revised: ( Y 1 , . . .

Original: , Y n ) can be written in terms of one-dimensional marginal distributions F i ( y ) =
Revised: , Y n ) can be written in terms of one-dimensional marginal distributions F i ( y ) =

Original: P ( Y i ≤ y ) and a copula C : [0 , 1] n → [0 , 1] that describes the dependence structures among variables:
Revised: P ( Y i ≤ y ) and a copula C : [0 , 1] n → [0 , 1] that describes the dependence structures among variables:

Original: [Equation]
Revised: [Equation]

Original: In other words, one can decompose a joint distribution into two components: the marginals and the copula.
Revised: In other words, one can decompose a joint distribution into two components: the marginals and the copula.

Original: Such decomposition allows a two-step approach to modeling a joint distribution: (1) learning the marginals F i ; (2) learning the copula C , where various parametric copula families are available.
Revised: Such decomposition allows a two-step approach to modeling a joint distribution: (1) learning the marginals F i ; (2) learning the copula C , where various parametric copula families are available.

Original: Furthermore, a copula C can also be regarded as the Cumulative Distribution Function (CDF) of a corresponding distribution on the unit hypercube [0 , 1] n .
Revised: Furthermore, a copula C can also be regarded as the Cumulative Distribution Function (CDF) of a corresponding distribution on the unit hypercube [0 , 1] n .

Original: Its copula density is denoted by c ( u 1 , . . .
Revised: Its copula density is denoted by c ( u 1 , . . .

Original: , u n ) :
Revised: , u n ) :

Original: = ∂ n C ( u 1 , u 2 , . . . , u n ) /∂u 1 · · · ∂u n .
Revised: = ∂ n C ( u 1 , u 2 , . . . , u n ) /∂u 1 · · · ∂u n .

Original: The Probability Density Function (PDF) of a random vector can be represented by its corresponding copula density.
Revised: The Probability Density Function (PDF) of a random vector can be represented by its corresponding copula density.

Original: If the random vector Y is continuous, its PDF can be written as
Revised: If the random vector Y is continuous, its PDF can be written as

Original: [Equation]
Revised: [Equation]

Original: where f i is the PDF of Y i , u i
Revised: where f i is the PDF of Y i , u i

Original: = F i ( y i ) , and c is the copula density.
Revised: = F i ( y i ) , and c is the copula density.

Original: For discrete random vectors, the form of the Probability Mass Function (PMF) is more complex.
Revised: For discrete random vectors, the form of the Probability Mass Function (PMF) is more complex.

Original: See Appendix B.2 for details.
Revised: See Appendix B.2 for details.

Original: Gaussian copula.
Revised: Gaussian copula.

Original: One of the most popular copula family is the Gaussian copula .
Revised: One of the most popular copula family is the Gaussian copula .

Original: When the joint distribution F is multivariate normal with a mean of 0 and a covariance matrix of Σ , the corresponding copula is the Gaussian copula:
Revised: When the joint distribution F is multivariate normal with a mean of 0 and a covariance matrix of Σ , the corresponding copula is the Gaussian copula:

Original: [Equation]
Revised: [Equation]

Original: where Φ n ( · ; 0 , R ) is the multivariate normal CDF, R is the correlation matrix of Σ , and Φ − 1 ( · ) is the quantile function of the univariate standard normal distribution.
Revised: where Φ n ( · ; 0 , R ) is the multivariate normal CDF, R is the correlation matrix of Σ , and Φ − 1 ( · ) is the quantile function of the univariate standard normal distribution.

Original: Its copula density is
Revised: Its copula density is

Original: [Equation]
Revised: [Equation]

Original: where I n is the identity matrix of size n and u = ( u 1 , u 2 , . .
Revised: where I n is the identity matrix of size n and u = ( u 1 , u 2 , . .

Original: . , u n ) .
Revised: . , u n ) .

Original: 4. T HE P ROPOSED M ODEL
Revised: 4. T HE P ROPOSED M ODEL

Original: Recall that our goal is to model both representational and correlational graph information in the conditional joint distribution of the node outcomes,
Revised: Recall that our goal is to model both representational and correlational graph information in the conditional joint distribution of the node outcomes,

Original: [Equation]
Revised: [Equation]

Original: which can be decomposed into the copula density and marginal densities.
Revised: which can be decomposed into the copula density and marginal densities.

Original: In this formulation, the representational information and correlational information are naturally separated into the marginal densities f
Revised: In this formulation, the representational information and correlational information are naturally separated into the marginal densities f

Original: i , for i = 1 , . . . , n , and the copula density c , respectively.
Revised: i , for i = 1 , . . . , n , and the copula density c , respectively.

Original: Note that both the marginal densities and the copula density are conditional on the node features X and the graph G .
Revised: Note that both the marginal densities and the copula density are conditional on the node features X and the graph G .

Original: Next, we need to choose a proper distribution family for each of these densities and parameterize the distribution parameters as functions of X and G .
Revised: Next, we need to choose a proper distribution family for each of these densities and parameterize the distribution parameters as functions of X and G .

Original: Choice of distribution family and parameterization for the copula density.
Revised: Choice of distribution family and parameterization for the copula density.

Original: For the distribution family, we choose the Gaussian copula as the copula family, c
Revised: For the distribution family, we choose the Gaussian copula as the copula family, c

Original: ( u 1 , . . . ,
Revised: ( u 1 , . . . ,

Original: u n ; Σ ( X , G ; θ )) , where the form of Σ ( · ; θ ) and the learnable parameters θ remain to be speciﬁed.
Revised: u n ; Σ ( X , G ; θ )) , where the form of Σ ( · ; θ ) and the learnable parameters θ remain to be speciﬁed.

Original: To leverage the correlational graph information, we draw a connection between the graph structure G and the covariance matrix Σ in the Gaussian copula density.
Revised: To leverage the correlational graph information, we draw a connection between the graph structure G and the covariance matrix Σ in the Gaussian copula density.

Original: Let K = Σ − 1 be the precision matrix; if two nodes i and j are not linked in the graph, we constrain the corresponding ( i, j ) -th entry in K to be 0 .
Revised: Let K = Σ − 1 be the precision matrix; if two nodes i and j are not linked in the graph, we constrain the corresponding ( i, j ) -th entry in K to be 0 .

Original: In other words, the absence of an edge between nodes i and j leads to their outcome variables y i and y j being conditionally independent given all other variables.
Revised: In other words, the absence of an edge between nodes i and j leads to their outcome variables y i and y j being conditionally independent given all other variables.

Original: The motivation of parameterizing the precision matrix K instead of the covariance matrix Σ is closely related to undirected graphical models (Lauritzen, 1996; Jordan et al., 2004; Wainwright & Jordan, 2008), where the conditional dependence structure among a set of variables is fully represented by edges in an underlying graph.
Revised: The motivation of parameterizing the precision matrix K instead of the covariance matrix Σ is closely related to undirected graphical models (Lauritzen, 1996; Jordan et al., 2004; Wainwright & Jordan, 2008), where the conditional dependence structure among a set of variables is fully represented by edges in an underlying graph.

Original: In our use case, we could view our assumption on K as a graphical model among random variables ( y 1 , . . .
Revised: In our use case, we could view our assumption on K as a graphical model among random variables ( y 1 , . . .

Original: , y n ) , where the underlying graph structure is known.
Revised: , y n ) , where the underlying graph structure is known.

Original: The conditional independence assumption has signiﬁcantly reduced the number of non-zero entries in K to be estimated.
Revised: The conditional independence assumption has signiﬁcantly reduced the number of non-zero entries in K to be estimated.

Original: However, without any further constraints, there are still |E| free parameters growing with the graph size, which can hardly be estimated accurately given only one observation of ( y 1 , . . .
Revised: However, without any further constraints, there are still |E| free parameters growing with the graph size, which can hardly be estimated accurately given only one observation of ( y 1 , . . .

Original: , y m ) .
Revised: , y m ) .

Original: In practice, we consider two ways of parametrizing K with fewer parameters.
Revised: In practice, we consider two ways of parametrizing K with fewer parameters.

Original: Two-parameter parametrization.
Revised: Two-parameter parametrization.

Original: A rather strong but simple constraint is to assume the non-zero off-diagonal entries of K have the same value or they are proportional to the corresponding entries in the normalized adjacency matrix, and introduce two global parameters controlling the overall strength of correlation.
Revised: A rather strong but simple constraint is to assume the non-zero off-diagonal entries of K have the same value or they are proportional to the corresponding entries in the normalized adjacency matrix, and introduce two global parameters controlling the overall strength of correlation.

Original: For example, we could have K = τ − 1 ( L + γ I ) as what we did in the simulation study in Section 3, or K =
Revised: For example, we could have K = τ − 1 ( L + γ I ) as what we did in the simulation study in Section 3, or K =

Original: β
Revised: β

Original: ( I n − α D − 1 / 2 AD − 1 / 2 ) as used in Jia & Benson (2020), where ( τ, γ ) or ( α, β ) are learnable parameters.
Revised: ( I n − α D − 1 / 2 AD − 1 / 2 ) as used in Jia & Benson (2020), where ( τ, γ ) or ( α, β ) are learnable parameters.

Original: Regression-based parametrization.
Revised: Regression-based parametrization.

Original: We further propose a more ﬂexible parameterization that allows the non-zero entries in K to be estimated by a regressor taking node features as inputs.
Revised: We further propose a more ﬂexible parameterization that allows the non-zero entries in K to be estimated by a regressor taking node features as inputs.

Original: In pariticular, for any ( i, j ) -pair corresponding to a non-zero entry of K , we set (cid:98)
Revised: In pariticular, for any ( i, j ) -pair corresponding to a non-zero entry of K , we set (cid:98)

Original: A i,j = softplus ( h ( x i , x j ; θ )) , where h is a two-layer MLP that takes the concatenation of x i and x j as input and outputs a scalar.
Revised: A i,j = softplus ( h ( x i , x j ; θ )) , where h is a two-layer MLP that takes the concatenation of x i and x j as input and outputs a scalar.

Original: Let (cid:98) D be the degree matrix if we treat (cid:98) A as a weighted adjacency matrix, and we set the precision matrix K
Revised: Let (cid:98) D be the degree matrix if we treat (cid:98) A as a weighted adjacency matrix, and we set the precision matrix K

Original: = I n + (cid:98) D − (cid:98) A .
Revised: = I n + (cid:98) D − (cid:98) A .

Original: This parameterization improves the ﬂexibility on estimating K while keeping the number of learnable parameters θ independent of the graph size n .
Revised: This parameterization improves the ﬂexibility on estimating K while keeping the number of learnable parameters θ independent of the graph size n .

Original: It also ensures that
Revised: It also ensures that

Original: K is positive-deﬁnite and thus invertible.
Revised: K is positive-deﬁnite and thus invertible.

Original: Choice of distribution families and parameterization for the marginal densities.
Revised: Choice of distribution families and parameterization for the marginal densities.

Original: One beneﬁt of the copula framework is the ﬂexibility on the choice of distribution families for the marginal densities.
Revised: One beneﬁt of the copula framework is the ﬂexibility on the choice of distribution families for the marginal densities.

Original: In this work, we choose the marginal densities to be normal distributions if the labels y are continuous variables, and we choose them to be Poisson distributions if the y are discrete.
Revised: In this work, we choose the marginal densities to be normal distributions if the labels y are continuous variables, and we choose them to be Poisson distributions if the y are discrete.

Original: We denote the i -th marginal density function by f i ( y i ; η i ( X , G ; θ )) and the corresponding CDF by
Revised: We denote the i -th marginal density function by f i ( y i ; η i ( X , G ; θ )) and the corresponding CDF by

Original: F i ( y i ; η i ( X , G ; θ )) , where η i ( · ; θ ) denotes the distribution parameters to be speciﬁed.
Revised: F i ( y i ; η i ( X , G ; θ )) , where η i ( · ; θ ) denotes the distribution parameters to be speciﬁed.

Original: If the i -th marginal distribution takes the form of a normal distribution N ( µ i , σ 2 i ) , then η i ( X , G ; θ ) =
Revised: If the i -th marginal distribution takes the form of a normal distribution N ( µ i , σ 2 i ) , then η i ( X , G ; θ ) =

Original: ( µ i ( X , G ; θ ) , σ 2 i ( X , G ; θ )) .
Revised: ( µ i ( X , G ; θ ) , σ 2 i ( X , G ; θ )) .

Original: We deﬁne µ i ( X , G ; θ ) as the output of a base GNN model for node i , and σ 2 i ( X , G ; θ ) as the i -th diagnoal element of the covariance matrix Σ ( X , G ; θ ) as we speciﬁed in the Gaussian copula.
Revised: We deﬁne µ i ( X , G ; θ ) as the output of a base GNN model for node i , and σ 2 i ( X , G ; θ ) as the i -th diagnoal element of the covariance matrix Σ ( X , G ; θ ) as we speciﬁed in the Gaussian copula.

Original: If the i -th marginal distribution takes the form of a Poisson distribution Pois( λ i ) , then η i ( X , G ; θ )
Revised: If the i -th marginal distribution takes the form of a Poisson distribution Pois( λ i ) , then η i ( X , G ; θ )

Original: =
Revised: =

Original: λ i ( X , G ; θ ) , and we deﬁne λ
Revised: λ i ( X , G ; θ ) , and we deﬁne λ

Original: i ( X , G ; θ ) as the output of a base GNN model for node i .
Revised: i ( X , G ; θ ) as the output of a base GNN model for node i .

Original: Either way, the representational role of the graph is reﬂected in the location parameters ( µ i or λ i ) computed by a base GNN model.
Revised: Either way, the representational role of the graph is reﬂected in the location parameters ( µ i or λ i ) computed by a base GNN model.

Original: In practice, we can also choose other distribution families such as the log-normal or negative binomial, depending on our belief on the true distributions of the node outcomes.
Revised: In practice, we can also choose other distribution families such as the log-normal or negative binomial, depending on our belief on the true distributions of the node outcomes.

Original: One can even choose different distribution families for different nodes simultaneously if necessary.
Revised: One can even choose different distribution families for different nodes simultaneously if necessary.

Original: 4. M ODEL L EARNING AND I NFERENCE
Revised: 4. M ODEL L EARNING AND I NFERENCE

Original: For simplicity of notation, we write η i ( X , G ; θ ) and Σ ( X , G ; θ ) as η i and Σ throughout this section.
Revised: For simplicity of notation, we write η i ( X , G ; θ ) and Σ ( X , G ; θ ) as η i and Σ throughout this section.

Original: Model learning.
Revised: Model learning.

Original: The model parameters θ are learned by maximizing the log-likelihood on the observed node labels.
Revised: The model parameters θ are learned by maximizing the log-likelihood on the observed node labels.

Original: Given the partition of y , we can further partition the covariance matrix Σ accordingly:
Revised: Given the partition of y , we can further partition the covariance matrix Σ accordingly:

Original: [Equation]
Revised: [Equation]

Original: where y obs =
Revised: where y obs =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y m ) and y miss =
Revised: , y m ) and y miss =

Original: ( y m +1 , . . .
Revised: ( y m +1 , . . .

Original: , y n ) .
Revised: , y n ) .

Original: In other words, Σ 00 and Σ 11 are the covariance matrices of the observed and missing nodes.
Revised: In other words, Σ 00 and Σ 11 are the covariance matrices of the observed and missing nodes.

Original: We further denote u i
Revised: We further denote u i

Original: = F i ( y i ; η i ) for i = 1 , . . . , n , u obs =
Revised: = F i ( y i ; η i ) for i = 1 , . . . , n , u obs =

Original: ( u 1 , . . .
Revised: ( u 1 , . . .

Original: , u m ) , and u miss = ( u m +1 , . . . , u n ) ; that is, u i is the probability integral transform of the i -th label y i .
Revised: , u m ) , and u miss = ( u m +1 , . . . , u n ) ; that is, u i is the probability integral transform of the i -th label y i .

Original: According to Equation (1), the joint density can be written as the product of the copula density and the marginal densities.
Revised: According to Equation (1), the joint density can be written as the product of the copula density and the marginal densities.

Original: Therefore the loss function, i.e., the negative log-likelihood, is
Revised: Therefore the loss function, i.e., the negative log-likelihood, is

Original: [Equation]
Revised: [Equation]

Original: The parameters θ are learned end-to-end using standard optimization algorithms such as
Revised: The parameters θ are learned end-to-end using standard optimization algorithms such as

Original: Adam (Kingma & Ba, 2015).
Revised: Adam (Kingma & Ba, 2015).

Original: Model inference.
Revised: Model inference.

Original: At inference time, we are interested in the conditional distribution f
Revised: At inference time, we are interested in the conditional distribution f

Original: ( y miss | y obs ; X , G ) .
Revised: ( y miss | y obs ; X , G ) .

Original: The inference of the conditional distribution can be done via sampling.
Revised: The inference of the conditional distribution can be done via sampling.

Original: Since f ( y ; X , G ) is modeled by the Gaussian copula, we have
Revised: Since f ( y ; X , G ) is modeled by the Gaussian copula, we have

Original: [Equation]
Revised: [Equation]

Original: where R is the correlation matrix corresponding to the covariance matrix Σ .
Revised: where R is the correlation matrix corresponding to the covariance matrix Σ .

Original: By the property of the multivatiate normal distribution, the conditional distribution of Φ − 1 ( u miss ) | Φ − 1 ( u obs ) is also multivatiate normal:
Revised: By the property of the multivatiate normal distribution, the conditional distribution of Φ − 1 ( u miss ) | Φ − 1 ( u obs ) is also multivatiate normal:

Original: [Equation]
Revised: [Equation]

Original: This provides a way to draw samples from f ( y miss | y obs ; X , G ) , which we describe in Algorithm 1.
Revised: This provides a way to draw samples from f ( y miss | y obs ; X , G ) , which we describe in Algorithm 1.

Original: Input:
Revised: Input:

Original: The node features X , the graph G , the observed node labels y 1 , . . .
Revised: The node features X , the graph G , the observed node labels y 1 , . . .

Original: , y m , the learned parameters θ , the marginal CDF functions F i ( · ; η i ( X , G ; θ ))
Revised: , y m , the learned parameters θ , the marginal CDF functions F i ( · ; η i ( X , G ; θ ))

Original: their inverse
Revised: their inverse

Original: F − 1 i ( · ; η i ( X , G ; θ )) for i = 1 , . . . , n , and the 
Revised: F − 1 i ( · ; η i ( X , G ; θ )) for i = 1 , . . . , n , and the number of samples do ← 15 return

Original: Scalability.
Revised: Scalability.

Original: Finally, we make a brief remark on the scalability of the proposed method.
Revised: Finally, we make a brief remark on the scalability of the proposed method.

Original: During model learning, the calculation of log c ( u obs ; Σ 00 ) in Eq.
Revised: During model learning, the calculation of log c ( u obs ; Σ 00 ) in Eq.

Original: (3) involves the log-determinant of the precision matrix K and its submatrix.
Revised: (3) involves the log-determinant of the precision matrix K and its submatrix.

Original: During model inference, both the conditional mean and variance involve the evaluation of R − 1 00 , but they can be transformed (via Schur complement) in a form that only requires solving a linear system with a submatrix of K as the coefﬁcients.
Revised: During model inference, both the conditional mean and variance involve the evaluation of R − 1 00 , but they can be transformed (via Schur complement) in a form that only requires solving a linear system with a submatrix of K as the coefﬁcients.

Original: Thanks to the sparse structure of K , both (one-step) learning and inference can be made efﬁcient with a computation cost linear in the graph size if K is well-conditioned.
Revised: Thanks to the sparse structure of K , both (one-step) learning and inference can be made efﬁcient with a computation cost linear in the graph size if K is well-conditioned.

Original: Jia & Benson (2020) provide an introduction of numerical techniques that can efﬁciently calculate the log-determinant (and its gradients) of a sparse matrix, as well as the solution of a linear system with sparse coefﬁcients, which can be directly applied to accelerate our proposed method.
Revised: Jia & Benson (2020) provide an introduction of numerical techniques that can efﬁciently calculate the log-determinant (and its gradients) of a sparse matrix, as well as the solution of a linear system with sparse coefﬁcients, which can be directly applied to accelerate our proposed method.

Original: In addition, as real-world graphs often present community structures, we can further improve the scalability of the proposed method by enforcing a block structure for the precision matrix.
Revised: In addition, as real-world graphs often present community structures, we can further improve the scalability of the proposed method by enforcing a block structure for the precision matrix.

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS 5. G ENERAL S ETUP
Revised:  E XPERIMENTS 5. G ENERAL S ETUP

Original: We instantiate CopulaGNN with either GCN or GraphSAGE as the base GNN models, and implement both the two-parameter parameterization (the ( α, β ) parameterization, denoted by “ αβ -C-”, where “C” stands for copula) and the regression-based parameterization (denoted by “R-C-”).
Revised: We instantiate CopulaGNN with either GCN or GraphSAGE as the base GNN models, and implement both the two-parameter parameterization (the ( α, β ) parameterization, denoted by “ αβ -C-”, where “C” stands for copula) and the regression-based parameterization (denoted by “R-C-”).

Original: In combination, we have four variants of CopulaGNN: αβ -C-GCN , R-C-GCN , αβ -C-SAGE , and RC-SAGE .
Revised: In combination, we have four variants of CopulaGNN: αβ -C-GCN , R-C-GCN , αβ -C-SAGE , and RC-SAGE .

Original: When the outcome is a continuous variable, the normal margin is used; and when the outcome is a count variable, the Poisson margin is used.
Revised: When the outcome is a continuous variable, the normal margin is used; and when the outcome is a count variable, the Poisson margin is used.

Original: In particular, in the former case, the αβ -CGNN degenerates to the Correlation GNN proposed by Jia & Benson (2020).
Revised: In particular, in the former case, the αβ -CGNN degenerates to the Correlation GNN proposed by Jia & Benson (2020).

Original: We compare different variants of CopulaGNN with their base GNN counterparts, as well as an MLP model, on two types of regression tasks: continuous outcome variables and count outcome variables.
Revised: We compare different variants of CopulaGNN with their base GNN counterparts, as well as an MLP model, on two types of regression tasks: continuous outcome variables and count outcome variables.

Original: More experiment details can be found in Appendix A.3.
Revised: More experiment details can be found in Appendix A.3.

Original: 5. R EGRESSION WITH C ONTINUOUS O UTCOME V ARIABLES
Revised: 5. R EGRESSION WITH C ONTINUOUS O UTCOME V ARIABLES

Original: We use two groups of datasets with continuous outcome variables.
Revised: We use two groups of datasets with continuous outcome variables.

Original: The ﬁrst group is the synthetic data of setting (c) as described in Sections 3.2 and 3.3, where a graph provides both representational and correlational information.
Revised: The ﬁrst group is the synthetic data of setting (c) as described in Sections 3.2 and 3.3, where a graph provides both representational and correlational information.

Original: The second group includes four regression tasks constructed from the
Revised: The second group includes four regression tasks constructed from the

Original: U.S. Election data (Jia & Benson, 2020).
Revised: U.S. Election data (Jia & Benson, 2020).

Original: We use the coefﬁcient of determination R 2 to measure the model performance.
Revised: We use the coefﬁcient of determination R 2 to measure the model performance.

Original: Results.
Revised: Results.

Original: For the synthetic datasets (Table 1), we vary the value of τ , which controls the overall magnitude of the label covariance.
Revised: For the synthetic datasets (Table 1), we vary the value of τ , which controls the overall magnitude of the label covariance.

Original: Unsurprisingly, as τ increases, the labels become noisier and the test R 2 of all models decreases.
Revised: Unsurprisingly, as τ increases, the labels become noisier and the test R 2 of all models decreases.

Original: In all conﬁgurations, R-C-GCN and R-C-SAGE respectively outperform their base model counterparts, GCN and SAGE, by signiﬁcant margins.
Revised: In all conﬁgurations, R-C-GCN and R-C-SAGE respectively outperform their base model counterparts, GCN and SAGE, by signiﬁcant margins.

Original: This veriﬁes the effectiveness of the proposed method when the graph provides both representational and correlational information.
Revised: This veriﬁes the effectiveness of the proposed method when the graph provides both representational and correlational information.

Original: Another interesting observation is that GCN outperforms MLP when τ is small (0.5 and 1.0), but underperforms MLP when τ becomes large (2.0 and 5.0), whereas R-CGCN consistently outperforms MLP.
Revised: Another interesting observation is that GCN outperforms MLP when τ is small (0.5 and 1.0), but underperforms MLP when τ becomes large (2.0 and 5.0), whereas R-CGCN consistently outperforms MLP.

Original: Note that τ can also be viewed as the tradeoff between the representational role and the correlational role served by the graph.
Revised: Note that τ can also be viewed as the tradeoff between the representational role and the correlational role served by the graph.

Original: The correlational role of the graph will have more inﬂuence on the outcome variables when τ becomes larger.
Revised: The correlational role of the graph will have more inﬂuence on the outcome variables when τ becomes larger.

Original: This explains the intriguing observation: GCN fails to utilize the correlational information and its advantages on the representational information diminish as τ increases.
Revised: This explains the intriguing observation: GCN fails to utilize the correlational information and its advantages on the representational information diminish as τ increases.

Original: For the U.S. Election dataset (Table 2), we observe that all variants of CopulaGNN signiﬁcantly outperform their base GNN counterparts.
Revised: For the U.S. Election dataset (Table 2), we observe that all variants of CopulaGNN signiﬁcantly outperform their base GNN counterparts.

Original: It is interesting that the simpler two-parameter parameterization outperforms the regression-based parameterization in most setups on this dataset.
Revised: It is interesting that the simpler two-parameter parameterization outperforms the regression-based parameterization in most setups on this dataset.

Original: One possible explanation is that the outcome variables that are connected in the graph tend to have strong correlations, since adjacent counties usually have similar statistics.
Revised: One possible explanation is that the outcome variables that are connected in the graph tend to have strong correlations, since adjacent counties usually have similar statistics.

Original: This is indeed suggested by Jia & Benson (2020).
Revised: This is indeed suggested by Jia & Benson (2020).

Original: The Unemployment task in particular, where the two-parameter parameterization appears to have the largest advantage, is shown to have the strongest correlation.
Revised: The Unemployment task in particular, where the two-parameter parameterization appears to have the largest advantage, is shown to have the strongest correlation.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 5. R EGRESSION WITH C OUNT O UTCOME V ARIABLES
Revised: 5. R EGRESSION WITH C OUNT O UTCOME V ARIABLES

Original: We use two groups of datasets with count outcome variables.
Revised: We use two groups of datasets with count outcome variables.

Original: The ﬁrst group consists of two
Revised: The ﬁrst group consists of two

Original: Wikipedia datasets: Wiki-Chameleon and Wiki-Squirrel (Rozemberczki et al., 2019); both are pagepage networks of Wikipedia pages with the visiting trafﬁc as node labels.
Revised: Wikipedia datasets: Wiki-Chameleon and Wiki-Squirrel (Rozemberczki et al., 2019); both are pagepage networks of Wikipedia pages with the visiting trafﬁc as node labels.

Original: The second group is a co-citation network of papers at the EMNLP conferences.
Revised: The second group is a co-citation network of papers at the EMNLP conferences.

Original: The goal is to predict the overall number of citations of each paper (including citations from outside EMNLP).
Revised: The goal is to predict the overall number of citations of each paper (including citations from outside EMNLP).

Original: We use the R 2 -deviance, an
Revised: We use the R 2 -deviance, an

Original: R 2 measure for count data (Cameron & Windmeijer, 1996)
Revised: R 2 measure for count data (Cameron & Windmeijer, 1996)

Original: [Table]
Revised: [Table]

Original: to measure the model performance.
Revised: to measure the model performance.

Original: Results.
Revised: Results.

Original: The results of the count regression tasks are shown in Table 3.
Revised: The results of the count regression tasks are shown in Table 3.

Original: Intuitively, hyper-linked web pages or co-cited papers are more likely to be visited or cited together, therefore leading to correlated outcome variables captured by the graph.
Revised: Intuitively, hyper-linked web pages or co-cited papers are more likely to be visited or cited together, therefore leading to correlated outcome variables captured by the graph.

Original: Indeed, we observe that the different variants of CopulaGNN outperform their base model counterparts in almost all setups.
Revised: Indeed, we observe that the different variants of CopulaGNN outperform their base model counterparts in almost all setups.

Original: However, as the correlation may not be as strong as in the U.S. Election dataset, we observe that the regressionbased parameterization (R-C-GCN and R-C-SAGE) has a greater advantage.
Revised: However, as the correlation may not be as strong as in the U.S. Election dataset, we observe that the regressionbased parameterization (R-C-GCN and R-C-SAGE) has a greater advantage.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we explicitly distinguish the representational and correlational roles of the graph representation of data.
Revised: In this work, we explicitly distinguish the representational and correlational roles of the graph representation of data.

Original: We demonstrate through a simulation study that many popular GNN models are incapable of fully utilizing the correlational graph information.
Revised: We demonstrate through a simulation study that many popular GNN models are incapable of fully utilizing the correlational graph information.

Original: Furthermore, we propose CopulaGNN, a principled method that improves upon a wide range of GNNs to achieve better prediction performance when the graph plays both representational and correlational roles.
Revised: Furthermore, we propose CopulaGNN, a principled method that improves upon a wide range of GNNs to achieve better prediction performance when the graph plays both representational and correlational roles.

Original: Compared with the corresponding base GNN models, multiple variants of CopulaGNN yield consistently superior results on both synthetic and real-world datasets for continuous and count regression tasks.
Revised: Compared with the corresponding base GNN models, multiple variants of CopulaGNN yield consistently superior results on both synthetic and real-world datasets for continuous and count regression tasks.

Original: A CKNOWLEDGEMENT
Revised: A CKNOWLEDGEMENT

Original: Jiaqi Ma and Qiaozhu Mei were in part supported by the National Science Foundation under grant numbers 1633370 and 1620319.
Revised: Jiaqi Ma and Qiaozhu Mei were in part supported by the National Science Foundation under grant numbers 1633370 and 1620319.

Original: A E XPERIMENT D ETAILS A. D ETAILS OF S YNTHETIC D ATA G ENERATION
Revised: A E XPERIMENT D ETAILS A. D ETAILS OF S YNTHETIC D ATA G ENERATION

Original: We generate the synthetic data with the following procedure.
Revised: We generate the synthetic data with the following procedure.

Original: Sample a node feature matrix X ∼ N ( 0 , I d 0 ) , where X ∈ R n × d 0 and d 0 is the feature dimension.
Revised: Sample a node feature matrix X ∼ N ( 0 , I d 0 ) , where X ∈ R n × d 0 and d 0 is the feature dimension.

Original: Generate the graph in a way similarly to a latent space model (Hoff et al., 2002).
Revised: Generate the graph in a way similarly to a latent space model (Hoff et al., 2002).

Original: First compute the latent variables Z = X W g , where W g ∈ R d 0 × d 1 is a given weight matrix.
Revised: First compute the latent variables Z = X W g , where W g ∈ R d 0 × d 1 is a given weight matrix.

Original: Then calculate the latent distance (cid:107) z i − z j (cid:107) 2 for each node pair ( i, j ) , 1 ≤ i < j ≤ n .
Revised: Then calculate the latent distance (cid:107) z i − z j (cid:107) 2 for each node pair ( i, j ) , 1 ≤ i < j ≤ n .

Original: Finally assign edges between the m pairs of nodes with the shortest latent distances to form a graph with n nodes and s edges.
Revised: Finally assign edges between the m pairs of nodes with the shortest latent distances to form a graph with n nodes and s edges.

Original: Assume A is the adjacency matrix of the graph, D is the degree matrix, and L = D − A is the graph Laplacian.
Revised: Assume A is the adjacency matrix of the graph, D is the degree matrix, and L = D − A is the graph Laplacian.

Original: Let ˜
Revised: Let ˜

Original: A
Revised: A

Original: =
Revised: =

Original: A + I and ˜ D
Revised: A + I and ˜ D

Original: = D
Revised: = D

Original: + I .
Revised: + I .

Original: Given parameters w y ∈ R d 0 , generate the node label vector y ∼ N ( µ , Σ ) , where, for some γ > 0 , τ > 0 , and σ 2 > 0 , (a) µ = ˜ D − 1 ˜
Revised: Given parameters w y ∈ R d 0 , generate the node label vector y ∼ N ( µ , Σ ) , where, for some γ > 0 , τ > 0 , and σ 2 > 0 , (a) µ = ˜ D − 1 ˜

Original: A X w y , Σ
Revised: A X w y , Σ

Original: = σ 2 I ; (b) µ = X w y , Σ
Revised: = σ 2 I ; (b) µ = X w y , Σ

Original: = τ ( L + γ I ) − 1 ; (c) µ =
Revised: = τ ( L + γ I ) − 1 ; (c) µ =

Original: ˜ D − 1 ˜
Revised: ˜ D − 1 ˜

Original: A X w y , Σ
Revised: A X w y , Σ

Original: = τ ( L + γ I ) − 1 .
Revised: = τ ( L + γ I ) − 1 .

Original: A. S IMULATION D ETAILS FOR S ECTION
Revised: A. S IMULATION D ETAILS FOR S ECTION

Original: For each conﬁguration we randomly generate 100 datasets with different seeds.
Revised: For each conﬁguration we randomly generate 100 datasets with different seeds.

Original: For each dataset, we randomly split the nodes into training, validation, and test sets equally to form a semi-supervised learning task.
Revised: For each dataset, we randomly split the nodes into training, validation, and test sets equally to form a semi-supervised learning task.

Original: We set the number of layers as 2 and the total hidden units as 16 for all models.
Revised: We set the number of layers as 2 and the total hidden units as 16 for all models.

Original: We use the Adam optimizer (Kingma & Ba, 2015) with an initial learning rate of 0.01 to train all models by minimizing the MSE loss, with early stopping on the validation set.
Revised: We use the Adam optimizer (Kingma & Ba, 2015) with an initial learning rate of 0.01 to train all models by minimizing the MSE loss, with early stopping on the validation set.

Original: Finally, we report the R 2 score on the test set.
Revised: Finally, we report the R 2 score on the test set.

Original: A. E XPERIMENT D ETAILS FOR S ECTION
Revised: A. E XPERIMENT D ETAILS FOR S ECTION

Original: Training details.
Revised: Training details.

Original: For all neural networks involved in the experiments, we set the number of layers as 2 and the number of hidden units as 16.
Revised: For all neural networks involved in the experiments, we set the number of layers as 2 and the number of hidden units as 16.

Original: We use the Adam optimizer to train all the models and apply early stopping on a validation set.
Revised: We use the Adam optimizer to train all the models and apply early stopping on a validation set.

Original: For the real-world datasets, the initial learning rate is chosen from { 0 . 01 , 0 .
Revised: For the real-world datasets, the initial learning rate is chosen from { 0 . 01 , 0 .

Original: 001 } on the validation set.
Revised: 001 } on the validation set.

Original: The U.S. Election dataset.
Revised: The U.S. Election dataset.

Original: The nodes in the election data are U.S. counties and edges connect adjacent counties on the map.
Revised: The nodes in the election data are U.S. counties and edges connect adjacent counties on the map.

Original: Each county is associated with demographic and election statistics.
Revised: Each county is associated with demographic and election statistics.

Original: In each of the regression tasks, one statistic is selected as the node outcome and the remaining statistics are used as the node features.
Revised: In each of the regression tasks, one statistic is selected as the node outcome and the remaining statistics are used as the node features.

Original: The four regression tasks are named by the outcome statistics:
Revised: The four regression tasks are named by the outcome statistics:

Original: Education, Election, Income, Unemployment.
Revised: Education, Election, Income, Unemployment.

Original: We randomly split the data into training, validation, and test sets with ratio 6:2:2 following Jia & Benson (2020), and we refer to their work for more details of the datasets.
Revised: We randomly split the data into training, validation, and test sets with ratio 6:2:2 following Jia & Benson (2020), and we refer to their work for more details of the datasets.

Original: The Wikipedia datasets.
Revised: The Wikipedia datasets.

Original: Each of the two Wikipedia datasets consists of a graph on Wikipedia, where each node is a Wikipedia page related to the animal of the page title and edges reﬂect mutual hyper-links between the pages.
Revised: Each of the two Wikipedia datasets consists of a graph on Wikipedia, where each node is a Wikipedia page related to the animal of the page title and edges reﬂect mutual hyper-links between the pages.

Original: The node features are principal components of binary indicators for the presence of certain nouns.
Revised: The node features are principal components of binary indicators for the presence of certain nouns.

Original: The count outcome variable of each node label is the monthly trafﬁc in the unit of thousands.
Revised: The count outcome variable of each node label is the monthly trafﬁc in the unit of thousands.

Original: The EMNLP dataset.
Revised: The EMNLP dataset.

Original: This dataset is constructed from the DBLP citation data provided by
Revised: This dataset is constructed from the DBLP citation data provided by

Original: AMiner (Tang et al., 2008).
Revised: AMiner (Tang et al., 2008).

Original: We ﬁrst extract a set of papers published on the EMNLP conference and treat each paper as a node.
Revised: We ﬁrst extract a set of papers published on the EMNLP conference and treat each paper as a node.

Original: Then we construct a graph where two papers have an edge if they are cited simultaneously by at least two EMNLP papers.
Revised: Then we construct a graph where two papers have an edge if they are cited simultaneously by at least two EMNLP papers.

Original: The node features are principal components of the bag-of-words of paper titles and abstracts as well as the year of publication.
Revised: The node features are principal components of the bag-of-words of paper titles and abstracts as well as the year of publication.

Original: The node label is the number of citations of each paper from outside EMNLP.
Revised: The node label is the number of citations of each paper from outside EMNLP.

Original: For both types of datasets, we randomly split the data into training, validation, and test sets with ratio 1:1:1.
Revised: For both types of datasets, we randomly split the data into training, validation, and test sets with ratio 1:1:1.

Original: [Figure]
Revised: [Figure]

Original: M ORE D ETAILS
Revised: M ORE D ETAILS

Original: [Figure]
Revised: [Figure]

Original: C
Revised: C

Original: [Figure]
Revised: [Figure]

Original: Figure 2 shows PDFs of two-dimensional distributions constructed using different parametric copulas; the marginal distributions are all standard normal.
Revised: Figure 2 shows PDFs of two-dimensional distributions constructed using different parametric copulas; the marginal distributions are all standard normal.

Original: B. A PPROXIMATING THE C OPULAS FOR D ISCRETE R ANDOM V ARIABLES
Revised: B. A PPROXIMATING THE C OPULAS FOR D ISCRETE R ANDOM V ARIABLES

Original: If the random vector Y is discrete, the copula representation of its PMF is more complex.
Revised: If the random vector Y is discrete, the copula representation of its PMF is more complex.

Original: Take n = 2 as an example, the PMF of Y =
Revised: Take n = 2 as an example, the PMF of Y =

Original: ( Y 1 , Y 2 ) is
Revised: ( Y 1 , Y 2 ) is

Original: [Equation]
Revised: [Equation]

Original: where u i 1 =
Revised: where u i 1 =

Original: lim x → y − i F i ( x ) =
Revised: lim x → y − i F i ( x ) =

Original: F i ( y − i ) and u i 2 =
Revised: F i ( y − i ) and u i 2 =

Original: F i ( y i ) .
Revised: F i ( y i ) .

Original: In general, the PMF has the following form:
Revised: In general, the PMF has the following form:

Original: [Equation]
Revised: [Equation]

Original: 5) which is computationally intractable because there are 2 n summands.
Revised: 5) which is computationally intractable because there are 2 n summands.

Original: Kazianka & Pilz (2010) propose an approximation of the above PMF based on the generalized quantile transform.
Revised: Kazianka & Pilz (2010) propose an approximation of the above PMF based on the generalized quantile transform.

Original: It smooths the CDF of ordinal discrete variables from a step function to a piece-wise linear one:
Revised: It smooths the CDF of ordinal discrete variables from a step function to a piece-wise linear one:

Original: [Equation]
Revised: [Equation]

Original: where v i =
Revised: where v i =

Original: ( u i 1 + u i 2 ) / 2 .
Revised: ( u i 1 + u i 2 ) / 2 .

Original: It has been shown that the approximation works well as long as the marginal variance is not too small.
Revised: It has been shown that the approximation works well as long as the marginal variance is not too small.

Original: We apply this method to approximate PMF to handle discrete random variables.
Revised: We apply this method to approximate PMF to handle discrete random variables.