Original: VEM
Revised: VEM

Original: GCN: T OPOLOGY O PTIMIZATION WITH V ARIATIONAL EM FOR G RAPH C ONVOLUTIONAL N ETWORKS
Revised: GCN: T OPOLOGY O PTIMIZATION WITH V ARIATIONAL EM FOR G RAPH C ONVOLUTIONAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Over-smoothing has emerged as a severe problem for node classiﬁcation with graph convolutional networks (GCNs).
Revised: Over-smoothing has emerged as a severe problem for node classiﬁcation with graph convolutional networks (GCNs).

Original: In the view of message passing, the oversmoothing issue is caused by the observed noisy graph topology that would propagate information along inter-class edges, and consequently, over-mix the features of nodes in different classes.
Revised: In the view of message passing, the oversmoothing issue is caused by the observed noisy graph topology that would propagate information along inter-class edges, and consequently, over-mix the features of nodes in different classes.

Original: In this paper , we propose a novel architecture, namely VEM-GCN to address this problem by employing the variational EM framework to jointly optimize the graph topology and learn desirable node representations for classiﬁcation.
Revised: In this paper, we propose a novel architecture, namely VEM-GCN, to address this problem by employing the variational EM algorithm to jointly optimize the graph topology and learn desirable node representations for classiﬁcation.

Original: Speciﬁcally variational EM approaches a latent adjacency matrix parameterized by the assortative-constrained stochastic block model (SBM) to enhance intra-class connection and suppress inter-class interaction of the observed noisy graph.
Revised: Speciﬁcally, variational EM approaches a latent adjacency matrix parameterized by the assortative-constrained stochastic block model (SBM) to enhance intra-class connection and suppress inter-class interaction of the observed noisy graph.

Original: In the variational E-step, graph topology is optimized by approximating the posterior probability distribution of the latent adjacency matrix with a neural network learned from node embeddings.
Revised: In the variational E-step, graph topology is optimized by approximating the posterior probability distribution of the latent adjacency matrix with a neural network learned from node embeddings.

Original: In the M-step, node representations are learned using the graph convolutional network based on the reﬁned graph topology for the downstream task of classiﬁcation.
Revised: In the M-step, node representations are learned using the graph convolutional network based on the reﬁned graph topology for the downstream task of classiﬁcation.

Original: VEM-GCN is demonstrated to outperform existing strategies for tackling over-smoothing and optimizing graph topology in node classiﬁcation on seven benchmark datasets.
Revised: VEM-GCN is demonstrated to outperform existing strategies for tackling over-smoothing and optimizing graph topology in node classiﬁcation on seven benchmark datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Complex graph-structured data are ubiquitous in the real world, ranging from social networks to chemical molecules.
Revised: Complex graph-structured data are ubiquitous in the real world, ranging from social networks to chemical molecules.

Original: Inspired by the remarkable performance of convolutional neural networks (CNNs) in processing data with regular grid structures (e.g., images), a myriad of studies on GCNs have emerged to execute “convolution” in the graph domain (Niepert et al., 2016; Kipf & Welling, 2017; Gilmer et al., 2017; Hamilton et al., 2017; Monti et al., 2017; Gao et al., 2018).
Revised: Inspired by the remarkable performance of convolutional neural networks (CNNs) in processing data with regular grid structures (e.g., images), a myriad of studies on GCNs have emerged to execute “convolution” in the graph domain (Niepert et al., 2016; Kipf & Welling, 2017; Gilmer et al., 2017; Hamilton et al., 2017; Monti et al., 2017; Gao et al., 2018).

Original: Many of these approaches follow a neighborhood aggregation mechanism (a.k.a. message passing scheme) that updates the representation of each node by iteratively aggregating the transformed messages sent from its neighboring nodes.
Revised: Many of these approaches follow a neighborhood aggregation mechanism (a.k.a., message passing scheme) that updates the representation of each node by iteratively aggregating the transformed messages sent from its neighboring nodes.

Original: Commencing with the pioneering works (Kipf & Welling, 2017;
Revised: Commencing with the pioneering works (Kipf & Welling, 2017;

Original: Gilmer et al., 2017), numerous strategies have been developed to improve the vanilla message passing scheme such as introducing self-attention mechanism (Veliˇckovi´c et al., 2018; Zhang et al., 2020), incorporating local structural information (Zhang et al., 2020;
Revised: Gilmer et al., 2017), numerous strategies have been developed to improve the vanilla message passing scheme such as introducing self-attention mechanism (Veliˇckovi´c et al., 2018; Zhang et al., 2020), incorporating local structural information (Zhang et al., 2020;

Original: Jin et al., 2019;
Revised: Jin et al., 2019;

Original: Ye et al., 2020), and leveraging the link attributes (Gong & Cheng, 2019; Li et al., 2019; Jiang et al., 2019).
Revised: Ye et al., 2020), and leveraging the link attributes (Gong & Cheng, 2019; Li et al., 2019; Jiang et al., 2019).

Original: Despite signiﬁcant success in many fundamental tasks of graph-based machine learning, message passing-based GCNs almost all process the observed graph structure as ground truth and might suffer from the over-smoothing problem (Li et al., 2018), which would seriously affect the node classiﬁcation performance.
Revised: Despite signiﬁcant success in many fundamental tasks of graph-based machine learning, message passing-based GCNs almost all process the observed graph structure as ground truth and might suffer from the over-smoothing problem (Li et al., 2018), which would seriously affect the node classiﬁcation performance.

Original: Given the observed noisy graph topology (i.e., excessive inter-class edges are linked while many intra-class edges are missing), when multiple message passing layers are stacked to enlarge the receptive ﬁeld (the maximum hop of neighborhoods), features of neighboring nodes in different classes would be dominant in message passing.
Revised: Given the observed noisy graph topology (i.e., excessive inter-class edges are linked while many intra-class edges are missing), when multiple message passing layers are stacked to enlarge the receptive ﬁeld (the maximum hop of neighborhoods), features of neighboring nodes in different classes would be dominant in message passing.

Original: Thus, node representations would be corrupted by the harmful noise and affect the discrimination of graph nodes.
Revised: Thus, node representations would be corrupted by the harmful noise and affect the discrimination of graph nodes.

Original: The over-smoothing phenomenon in GCNs has already been studied from different aspects.
Revised: The over-smoothing phenomenon in GCNs has already been studied from different aspects.

Original: Li et al.
Revised: Li et al.

Original: ﬁrst interpreted over-smoothing from the perspective of Laplacian smoothing, while Xu et al.
Revised: ﬁrst interpreted over-smoothing from the perspective of Laplacian smoothing, while Xu et al.

Original: and
Revised: and

Original: Klicpera et al.
Revised: Klicpera et al.

Original: (2019a) associated it with the limit distribution of random walk.
Revised: (2019a) associated it with the limit distribution of random walk.

Original: Furthermore, Chen et al.
Revised: Furthermore, Chen et al.

Original: (2020a) developed quantitative metrics to measure the over-smoothness from the topological view.
Revised: (2020a) developed quantitative metrics to measure the over-smoothness from the topological view.

Original: They argued that the key factor leading to over-smoothing is the noise passing between nodes of different categories and the classiﬁcation performance of GCNs is positively correlated with the proportion of intra-class node pairs in all edges.
Revised: They argued that the key factor leading to over-smoothing is the noise passing between nodes of different categories and the classiﬁcation performance of GCNs is positively correlated with the proportion of intra-class node pairs in all edges.

Original: In this paper, we propose VEM-GCN, a novel architecture to address the over-smoothing problem with topology optimization for uncertain graphs.
Revised: In this paper, we propose VEM-GCN, a novel architecture to address the over-smoothing problem with topology optimization for uncertain graphs.

Original: Considering that a “clearer” graph with more intraclass edges and fewer inter-class edges would improve the node classiﬁcation performance of GCNs (Yang et al., 2019; Chen et al., 2020a), VEM-GCN approaches a latent adjacency matrix parameterized by an assortative-constrained stochastic block model (SBM) where nodes share the same label are linked and inter-class edges should be cut off.
Revised: Considering that a “clearer” graph with more intraclass edges and fewer inter-class edges would improve the node classiﬁcation performance of GCNs (Yang et al., 2019; Chen et al., 2020a), VEM-GCN approaches a latent adjacency matrix parameterized by an assortative-constrained stochastic block model (SBM) where nodes share the same label are linked and inter-class edges should be cut off.

Original: To jointly reﬁne the latent graph structure and learn desirable node representations for classiﬁcation, variational EM algorithm is adopted to optimize the evidence lower bound (ELBO) of the likelihood function.
Revised: To jointly reﬁne the latent graph structure and learn desirable node representations for classiﬁcation, variational EM algorithm is adopted to optimize the evidence lower bound (ELBO) of the likelihood function.

Original: In the inference procedure (E-step), graph topology is optimized by approximating the posterior probability distribution of the latent adjacency matrix with a neural network learned from node embeddings.
Revised: In the inference procedure (E-step), graph topology is optimized by approximating the posterior probability distribution of the latent adjacency matrix with a neural network learned from node embeddings.

Original: In the learning procedure (M-step), a conventional GCN is trained to maximize the log-likelihood of the observed node labels based on the learned latent graph structure.
Revised: In the learning procedure (M-step), a conventional GCN is trained to maximize the log-likelihood of the observed node labels based on the learned latent graph structure.

Original: The E-step and M-step optimize the graph topology and improve the classiﬁcation of unlabeled nodes in an alternating fashion.
Revised: The E-step and M-step optimize the graph topology and improve the classiﬁcation of unlabeled nodes in an alternating fashion.

Original: The proposed VEM-GCN architecture is ﬂexible and general.
Revised: The proposed VEM-GCN architecture is ﬂexible and general.

Original: In the E-step, the neural network can support arbitrary desirable node embeddings generated by algorithms such as node2vec (Grover &
Revised: In the E-step, the neural network can support arbitrary desirable node embeddings generated by algorithms such as node2vec (Grover &

Original: Leskovec, 2016), struc2vec (Ribeiro et al., 2017), and GCNs, or the raw node attributes.
Revised: Leskovec, 2016), struc2vec (Ribeiro et al., 2017), and GCNs, or the raw node attributes.

Original: The GCN in the M-step can also be substituted with arbitrary graph models.
Revised: The GCN in the M-step can also be substituted with arbitrary graph models.

Original: Furthermore, recent strategies for relieving the over-smoothing issue, i.e., AdaEdge (Chen et al., 2020a) and DropEdge (Rong et al., 2020), are shown to be the speciﬁc cases of VEM-GCN under certain conditions.
Revised: Furthermore, recent strategies for relieving the over-smoothing issue, i.e., AdaEdge (Chen et al., 2020a) and DropEdge (Rong et al., 2020), are shown to be the speciﬁc cases of VEM-GCN under certain conditions.

Original: For empirical evaluation, we conduct extensive experiments on seven benchmarks for node classiﬁcation, including four citation networks, two Amazon co-purchase graphs, and one Microsoft Academic graph.
Revised: For empirical evaluation, we conduct extensive experiments on seven benchmarks for node classiﬁcation, including four citation networks, two Amazon co-purchase graphs, and one Microsoft Academic graph.

Original: Experimental results demonstrate the effectiveness of the proposed VEM-GCN architecture in optimizing graph topology and mitigating the over-smoothing problem for GCNs.
Revised: Experimental results demonstrate the effectiveness of the proposed VEM-GCN architecture in optimizing graph topology and mitigating the over-smoothing problem for GCNs.

Original:  B ACKGROUND AND R ELATED W ORKS
Revised:  B ACKGROUND AND R ELATED W ORKS

Original: Problem Setting.
Revised: Problem Setting.

Original: This paper focuses on the task of graph-based transductive node classiﬁcation.
Revised: This paper focuses on the task of graph-based transductive node classiﬁcation.

Original: A simple attributed graph is deﬁned as a tuple G obs =
Revised: A simple attributed graph is deﬁned as a tuple G obs =

Original: ( V , A obs , X ) , where V = { v i } Ni =1 is the node set, A obs = (cid:2) a obs ij (cid:3) ∈ { 0 , 1 } N × N is the observed adjacency matrix, and X ∈ R N × f represents the collection of attributes with each row corresponding to the features of an individual node.
Revised: ( V , A obs , X ) , where V = { v i } Ni =1 is the node set, A obs = (cid:2) a obs ij (cid:3) ∈ { 0 , 1 } N × N is the observed adjacency matrix, and X ∈ R N × f represents the collection of attributes with each row corresponding to the features of an individual node.

Original: Given the labels Y l =
Revised: Given the labels Y l =

Original: [ y ic ] ∈ { 0 , 1 } |V l |× C for a subset of graph nodes V l ⊂ V assigned to C classes, the task is to infer the classes Y u
Revised: [ y ic ] ∈ { 0 , 1 } |V l |× C for a subset of graph nodes V l ⊂ V assigned to C classes, the task is to infer the classes Y u

Original: =
Revised: =

Original: [ y jc ] ∈ { 0 , 1 } |V u |× C for the unlabeled nodes V u = V\V l based on G obs .
Revised: [ y jc ] ∈ { 0 , 1 } |V u |× C for the unlabeled nodes V u = V\V l based on G obs .

Original: Graph Convolutional Networks (GCNs).
Revised: Graph Convolutional Networks (GCNs).

Original: The core of most GCNs is message passing scheme, where each node updates its representation by iteratively aggregating features from its neighborhoods.
Revised: The core of most GCNs is message passing scheme, where each node updates its representation by iteratively aggregating features from its neighborhoods.

Original: Denote with W ( l ) the learnable weights in the l -th layer, N ( i ) the set of neighboring node indices for node v i , and σ ( · ) the nonlinear activation function.
Revised: Denote with W ( l ) the learnable weights in the l -th layer, N ( i ) the set of neighboring node indices for node v i , and σ ( · ) the nonlinear activation function.

Original: A message passing layer follows
Revised: A message passing layer follows

Original: [Equation]
Revised: [Equation]

Original: Here, h ( l ) j is the input features of node v j in the l -th layer, W l h (
Revised: Here, h ( l ) j is the input features of node v j in the l -th layer, W l h (

Original: [Equation]
Revised: [Equation]

Original: ) j is the corresponding transformed message, and α ij is the aggregation weight for the message passing from node v j to node v i .
Revised: ) j is the corresponding transformed message, and α ij is the aggregation weight for the message passing from node v j to node v i .

Original: Existing GCNs mainly differ in the mechanism for computing α ij (Kipf & Welling, 2017; Veliˇckovi´c et al., 2018;
Revised: Existing GCNs mainly differ in the mechanism for computing α ij (Kipf & Welling, 2017; Veliˇckovi´c et al., 2018;

Original: Ye et al., 2020; Hamilton et al., 2017; Zhang et al., 2020).
Revised: Ye et al., 2020; Hamilton et al., 2017; Zhang et al., 2020).

Original: Stochastic Block Model (SBM).
Revised: Stochastic Block Model (SBM).

Original: SBM (Holland et al., 1983) is a generative model for producing graphs with community structures.
Revised: SBM (Holland et al., 1983) is a generative model for producing graphs with community structures.

Original: It parameterizes the edge probability between each node pair by
Revised: It parameterizes the edge probability between each node pair by

Original: [Equation]
Revised: [Equation]

Original: where ¯ a ij is an indicator variable for the edge linking nodes v i and v j , y i and y j denote their corresponding communities (classes), p 0 and p 1 are termed community link strength and crosscommunity link probability, respectively.
Revised: where ¯ a ij is an indicator variable for the edge linking nodes v i and v j , y i and y j denote their corresponding communities (classes), p 0 and p 1 are termed community link strength and crosscommunity link probability, respectively.

Original: The case where p 0 > p 1 is called an assortative model, while the case p 0 < p 1 is called disassortative.
Revised: The case where p 0 > p 1 is called an assortative model, while the case p 0 < p 1 is called disassortative.

Original: In this paper, we leverage an assortative-constrained
Revised: In this paper, we leverage an assortative-constrained

Original: SBM (Gribel et al., 2020) with p 0 = 1 and p 1 = 0 to model the latent graph for a clear topology.
Revised: SBM (Gribel et al., 2020) with p 0 = 1 and p 1 = 0 to model the latent graph for a clear topology.

Original: Over-smoothing.
Revised: Over-smoothing.

Original: Real-world graphs often possess high sparsity and are corrupted by certain noise that leads to inter-class misconnection and missing intra-class edges.
Revised: Real-world graphs often possess high sparsity and are corrupted by certain noise that leads to inter-class misconnection and missing intra-class edges.

Original: Over-smoothing is mainly caused by the indistinguishable features of nodes in different classes produced by the message passing along inter-class edges.
Revised: Over-smoothing is mainly caused by the indistinguishable features of nodes in different classes produced by the message passing along inter-class edges.

Original: Various strategies have been developed to alleviate this problem.
Revised: Various strategies have been developed to alleviate this problem.

Original: JK-Net (Xu et al., 2018) utilizes skip connection for adaptive feature aggregation and DNA (Fey, 2019) further makes improvements based on the attention mechanism.
Revised: JK-Net (Xu et al., 2018) utilizes skip connection for adaptive feature aggregation and DNA (Fey, 2019) further makes improvements based on the attention mechanism.

Original: PPNP and APPNP (Klicpera et al., 2019a) modify the message passing scheme by personalized PageRank (PPR) to avoid reaching the limit distribution of random walk.
Revised: PPNP and APPNP (Klicpera et al., 2019a) modify the message passing scheme by personalized PageRank (PPR) to avoid reaching the limit distribution of random walk.

Original: CGNN (Xhonneux et al., 2020) addresses over-smoothing in a similar manner as PPR.
Revised: CGNN (Xhonneux et al., 2020) addresses over-smoothing in a similar manner as PPR.

Original: Zhao & Akoglu (2020) introduced a graph layer normalization scheme termed PairNorm to maintain the total pairwise distance between nodes unchanged across layers.
Revised: Zhao & Akoglu (2020) introduced a graph layer normalization scheme termed PairNorm to maintain the total pairwise distance between nodes unchanged across layers.

Original: GCNII (Chen et al., 2020b) extends GCN with initial residual and identity mapping.
Revised: GCNII (Chen et al., 2020b) extends GCN with Initial residual and Identity mapping.

Original: However, these methods cannot well address the over-smoothing issue, as they all view the observed graph as ground truth and the features of nodes from different clusters would still be over-mixed along the inter-class edges.
Revised: However, these methods cannot well address the over-smoothing issue, as they all view the observed graph as ground truth and the features of nodes in different classes would still be over-mixed along the inter-class edges.

Original: AdaEdge (Chen et al., 2020a) constantly reﬁnes the graph topology by adjusting the edges in a self-training-like fashion.
Revised: AdaEdge (Chen et al., 2020a) constantly reﬁnes the graph topology by adjusting the edges in a self-training-like fashion.

Original: However, AdaEdge only adjusts the edges linking nodes classiﬁed with high conﬁdence, which leads to limited improvement or degradation in classiﬁcation performance due to the incorrect operations between misclassiﬁed nodes.
Revised: However, AdaEdge only adjusts the edges linking nodes classiﬁed with high conﬁdence, which leads to limited improvement or degradation in classiﬁcation performance due to the incorrect operations for misclassiﬁed nodes.

Original: DropEdge (Rong et al., 2020) randomly removes a certain fraction of edges to reduce message passing.
Revised: DropEdge (Rong et al., 2020) randomly removes a certain fraction of edges to reduce message passing.

Original: Despite enhanced robustness, DropEdge does not essentially optimize the graph topology.
Revised: Despite enhanced robustness, DropEdge does not essentially optimize the graph topology.

Original: BBGDC (Hasanzadeh et al., 2020) generalizes Dropout (Srivastava et al., 2014) and DropEdge by adaptive connection sampling.
Revised: BBGDC (Hasanzadeh et al., 2020) generalizes Dropout (Srivastava et al., 2014) and DropEdge by adaptive connection sampling.

Original: Uncertain Graphs and Topology Optimization.
Revised: Uncertain Graphs and Topology Optimization.

Original: Learning with uncertain graphs is another related research area, where the observed graph structure is supposed to be derived from noisy data rather than ground truth.
Revised: Learning with uncertain graphs is another related research area, where the observed graph structure is supposed to be derived from noisy data rather than ground truth.

Original: Bayesian approaches are typical methods that introduce uncertainty to network analysis.
Revised: Bayesian approaches are typical methods that introduce uncertainty to network analysis.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2019) developed BGCN that considers the observed graph as a sample from a parametric family of random graphs and makes maximum a posteriori (MAP) estimate of the graph parameters.
Revised: (2019) developed BGCN that considers the observed graph as a sample from a parametric family of random graphs and makes maximum a posteriori (MAP) estimate of the graph parameters.

Original: Tiao et al.
Revised: Tiao et al.

Original: (2019) also viewed graph edges as Bernoulli random variables and used variational inference to optimize the posterior distribution of the adjacency matrix by approximating the pre-deﬁned graph priors.
Revised: (2019) also viewed graph edges as Bernoulli random variables and used variational inference to optimize the posterior distribution of the adjacency matrix by approximating the pre-deﬁned graph priors.

Original: Some other Bayesian methods have also been developed to combine GCNs with probabilistic models (Ng et al., 2018; Ma et al., 2019).
Revised: Some other Bayesian methods have also been developed to combine GCNs with probabilistic models (Ng et al., 2018; Ma et al., 2019).

Original: However, without explicit optimization to the graph structure, they only improve the robustness under certain conditions such as incomplete edges, active learning, and adversarial attacks.
Revised: However, without explicit optimization for the graph structure, they only improve the robustness under certain conditions such as incomplete edges, active learning, and adversarial attacks.

Original: For explicit topology optimization, Franceschi et al.
Revised: For explicit topology optimization, Franceschi et al.

Original: (2019) presented LDS to parameterize edges as independent Bernoulli random variables and learn discrete structures for GCNs by solving a bilevel programming.
Revised: (2019) presented LDS to parameterize edges as independent Bernoulli random variables and learn discrete structures for GCNs by solving a bilevel programming.

Original: However, LDS requires an extra validation set for training and suffers from limited scalability.
Revised: However, LDS requires an extra validation set for training and suffers from limited scalability.

Original: TO-GCN (Yang et al., 2019) only adds the intra-class edges derived from the labeled nodes, which causes topology imbalance between V u and V l .
Revised: TO-GCN (Yang et al., 2019) only adds the intra-class edges derived from the labeled nodes, which causes topology imbalance between V u and V l .

Original: GDC (Klicpera et al., 2019b) reﬁnes the adjacency matrix with graph diffusion to consider the links between high-order neighborhoods.
Revised: GDC (Klicpera et al., 2019b) reﬁnes the adjacency matrix with graph diffusion to consider the links between high-order neighborhoods.

Original: However, the added edges might still be noisy to hamper the classiﬁcation.
Revised: However, the added edges might still be noisy to hamper the classiﬁcation.

Original: Different from the aforementioned methods, VEM-GCN introduces a learned latent graph based on the assortative-constrained SBM to  enhance intra-class connection and suppress inter-class interaction with variational EM to relieve the over-smoothing issue.
Revised: Different from the aforementioned methods, VEM-GCN relieves over-smoothing by introducing a learned latent graph based on the assortative-constrained SBM to explicitly enhance intra-class connection and suppress inter-class interaction with variational EM.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In this section, we develop the VEM-GCN architecture for transductive node classiﬁcation.
Revised: In this section, we develop the VEM-GCN architecture for transductive node classiﬁcation.

Original: VEMGCN leverages the variational EM algorithm to achieve topology optimization, and consequently, address the over-smoothing problem by reducing noisy interactions between nodes in different classes.
Revised: VEMGCN leverages the variational EM algorithm to achieve topology optimization, and consequently, address the over-smoothing problem by reducing noisy interactions between nodes in different classes.

Original: Speciﬁcally, E-step approximates the posterior probability distribution of the latent adjacency matrix to optimize the graph structure, and M-step maximizes the evidence lower bound of the log-likelihood function based on the reﬁned graph.
Revised: Speciﬁcally, E-step approximates the posterior probability distribution of the latent adjacency matrix to optimize the graph structure, and M-step maximizes the evidence lower bound of the log-likelihood function based on the reﬁned graph.

Original: We ﬁrst introduce our motivation and provide an overview of the proposed VEM-GCN architecture.
Revised: We ﬁrst introduce our motivation and provide an overview of the proposed VEM-GCN architecture.

Original: Subsequently, we elaborate the mechanisms of the variational E-step and M-step, respectively.
Revised: Subsequently, we elaborate the mechanisms of the variational E-step and M-step, respectively.

Original: 3. M OTIVATION AND O VERVIEW
Revised: 3. M OTIVATION AND O VERVIEW

Original: Motivation.
Revised: Motivation.

Original: As mentioned above, a graph with its nodes densely connected within their own communities (classes) has lower risk of over-smoothing.
Revised: As mentioned above, a graph with its nodes densely connected within their own communities (classes) has lower risk of over-smoothing.

Original: Under this consideration, the optimal adjacency matrix for GCN is ˜ A
Revised: Under this consideration, the optimal adjacency matrix for GCN is ˜ A

Original: =
Revised: =

Original: YY (cid:62) (Yang et al., 2019; Chen et al., 2020a), where Y ∈ R N × C is the matrix of one-hot-encoded ground-truth labels.
Revised: YY (cid:62) (Yang et al., 2019; Chen et al., 2020a), where Y ∈ R N × C is the matrix of one-hot-encoded ground-truth labels.

Original: However, since we have to infer Y u for the unlabeled nodes V u , their true labels are not available for calculating ˜ A .
Revised: However, since we have to infer Y u for the unlabeled nodes V u , their true labels are not available for calculating ˜ A .

Original: Thus, we introduce a latent graph A latent learned from G obs with another auxiliary neural network that helps generate a topology clearer than A obs for GCNs.
Revised: Thus, we introduce a latent graph A latent learned from G obs with another auxiliary neural network that helps generate a topology clearer than A obs for GCNs.

Original: It is obvious that ˜ A is equivalent to a SBM with p 0 = 1 and p 1 = 0 , and therefore we base the posterior probability distribution of the latent graph on this assumption.
Revised: It is obvious that ˜ A is equivalent to a SBM with p 0 = 1 and p 1 = 0 , and therefore we base the posterior probability distribution of the latent graph on this assumption.

Original: Overview.
Revised: Overview.

Original: The basic principle behind our proposed VEM-GCN architecture is maximum likelihood estimation (MLE) in a latent variable model, i.e., to maximize the log-likelihood function of the observed node labels E q φ ( A latent |G obs ) [log p θ ( Y l |G obs )] based on the approximate posterior distribution q φ ( A latent |G obs ) of the latent graph A latent .
Revised: The basic principle behind our proposed VEM-GCN architecture is maximum likelihood estimation (MLE) in a latent variable model, i.e., to maximize the log-likelihood function of the observed node labels E q φ ( A latent |G obs ) [log p θ ( Y l |G obs )] based on the approximate posterior distribution q φ ( A latent |G obs ) of the latent graph A latent .

Original: According to variational inference, the evidence lower bound (ELBO) is optimized instead:
Revised: According to variational inference, the evidence lower bound (ELBO) is optimized instead:

Original: [Equation]
Revised: [Equation]

Original: where the equality holds when q φ ( A latent |G obs )
Revised: where the equality holds when q φ ( A latent |G obs )

Original: = p θ ( A latent | Y l , G obs ) .
Revised: = p θ ( A latent | Y l , G obs ) .

Original: Note that q φ can be arbitrary desirable distributions on A latent and we use a neural network to parameterize it in this work.
Revised: Note that q φ can be arbitrary desirable distributions on A latent and we use a neural network to parameterize it in this work.

Original: To jointly optimize the latent graph topology A latent and the ELBO L ELBO ( θ, φ ; Y l , G obs ) , we adopt the variational EM framework to solve it (refer to Appendix A for the full algorithm).
Revised: To jointly optimize the latent graph topology A latent and the ELBO L ELBO ( θ, φ ; Y l , G obs ) , we adopt the variational EM algorithm to solve it (refer to Appendix A for the full algorithm).

Original: 3. E- STEP
Revised: 3. E- STEP

Original: In the inference procedure (E-step), θ is ﬁxed and the goal is to optimze q φ ( A latent |G obs ) to approximate the true posterior distribution p θ ( A latent | Y l , G obs ) .
Revised: In the inference procedure (E-step), θ is ﬁxed and the goal is to optimze q φ ( A latent |G obs ) to approximate the true posterior distribution p θ ( A latent | Y l , G obs ) .

Original: Under the condition of SBM, we assume each edge of the latent graph to be independent.
Revised: Under the condition of SBM, we assume each edge of the latent graph to be independent.

Original: Thus, q φ ( A latent |G obs ) can be factorized by:
Revised: Thus, q φ ( A latent |G obs ) can be factorized by:

Original: [Equation]
Revised: [Equation]

Original: Unlike LDS (Franceschi et al., 2019) using  N 2  Bernoulli random variables to characterize the optimized graph with N nodes, we parameterize q φ ( a latent ij |G obs ) through a neural network shared by all the possible node pairs (i.e., amortized variational inference (Gershman & Goodman, 2014)), as shown in Eq.
Revised: Unlike LDS (Franceschi et al., 2019) using O ( N 2 ) Bernoulli random variables to characterize the optimized graph with N nodes, we parameterize q φ ( a latent ij |G obs ) through a neural network shared by all the possible node pairs (i.e., amortized variational inference (Gershman & Goodman, 2014)), as shown in Eq.

Original: Hence, our method shows scalability for large-scale graphs and is easier to train.
Revised: Hence, our method shows scalability for large-scale graphs and is easier to train.

Original: [Equation]
Revised: [Equation]

Original: where e i is the node embedding of node v i , which can be derived from any desirable network embedding methods (e.g., node2vec (Grover & Leskovec, 2016), struc2vec
Revised: where e i is the node embedding of node v i , which can be derived from any desirable network embedding methods (e.g., node2vec (Grover & Leskovec, 2016), struc2vec

Original: (Ribeiro et al., 2017), and GCNs) or the raw node attributes x i (the i -th row of X ), and z i is the transformed features of node v i .
Revised: (Ribeiro et al., 2017), and GCNs) or the raw node attributes x i (the i -th row of X ), and z i is the transformed features of node v i .

Original: NN ( · ) denotes a neural network and we use a Multi-Layer Perceptron (MLP) in this work.
Revised: NN ( · ) denotes a neural network and we use a Multi-Layer Perceptron (MLP) in this work.

Original: The probability for linking a node pair is deﬁned as the inner-product of their transformed features activated by a sigmoid function.
Revised: The probability for linking a node pair is deﬁned as the inner-product of their transformed features activated by a sigmoid function.

Original: To approximate the posterior probability distribution of A latent , we rewrite p θ ( A latent | Y l , G obs ) as:
Revised: To approximate the posterior probability distribution of A latent , we rewrite p θ ( A latent | Y l , G obs ) as:

Original: [Equation]
Revised: [Equation]

Original:  obs )] .
Revised: , G obs )] .

Original: [Equation]
Revised: [Equation]

Original: Here, p θ ( A latent | Y l , Y u , G obs ) is parameterized by the aforementioned assortative-constrained SBM (i.e., p θ ( a latent ij =
Revised: Here, p θ ( A latent | Y l , Y u , G obs ) is parameterized by the aforementioned assortative-constrained SBM (i.e., p θ ( a latent ij =

Original: [Equation]
Revised: [Equation]

Original: | y i , y
Revised: y i , y

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: y i y (cid:62) j for the one-hot-encoded node label y ), p θ ( Y u | Y l , G obs ) is the predicted categorical distributions for the unlabeled nodes derived in the previous M-step.
Revised: y i y (cid:62) j for the one-hot-encoded node label y ), p θ ( Y u | Y l , G obs ) is the predicted categorical distributions for the unlabeled nodes derived in the previous M-step.

Original: Consequently, we can sample ˆ Y u ∼ p θ ( Y u | Y l , G obs ) to estimate the expectation in Eq.
Revised: Consequently, we can sample ˆ Y u ∼ p θ ( Y u | Y l , G obs ) to estimate the expectation in Eq.

Original: 6 and leverage stochastic gradient descent (SGD) to minimize the reverse KL-divergence between the approximate posterior distribution q φ ( A latent |G obs ) and the target p θ
Revised: 6 and leverage stochastic gradient descent (SGD) to minimize the reverse KL-divergence between the approximate posterior distribution q φ ( A latent |G obs ) and the target p θ

Original: ( A latent | Y l , G obs ) .
Revised: ( A latent | Y l , G obs ) .

Original: Under appropriate assumptions, q φ will converge to p θ ( A latent | Y l , G obs ) as the iteration step of SGD t → ∞ (Bottou, 2010).
Revised: Under appropriate assumptions, q φ will converge to p θ ( A latent | Y l , G obs ) as the iteration step of SGD t → ∞ (Bottou, 2010).

Original: Thus, we can obtain the following objective function in the variational E-step for optimizing φ :
Revised: Thus, we can obtain the following objective function in the variational E-step for optimizing φ :

Original: [Equation]
Revised: [Equation]

Original: where y is the ground truth label for node in labeled set V l , otherwise sampled from p
Revised: where y is the ground truth label for node in labeled set V l , otherwise sampled from p

Original: θ ( Y u | Y l , G obs ) for the nodes without given labels in each training step, and λ ( a latent ij ) is the weighting hyperparameter to alleviate class imbalance between the inter-class edges and the intra-class edges.
Revised: θ ( Y u | Y l , G obs ) for the nodes without given labels in each training step, and λ ( a latent ij ) is the weighting hyperparameter to alleviate class imbalance between the inter-class edges and the intra-class edges.

Original: 3. M- STEP
Revised: 3. M- STEP

Original: In the learning procedure (M-step), φ is ﬁxed and θ is updated to maximize the ELBO in Eq.
Revised: In the learning procedure (M-step), φ is ﬁxed and θ is updated to maximize the ELBO in Eq.

Original: By factorizing p θ ( Y l , A latent |G obs ) = p
Revised: By factorizing p θ ( Y l , A latent |G obs ) = p

Original: [Equation]
Revised: [Equation]

Original: 1 ( Y l | A latent , G obs ) p θ 2 ( A latent |G obs ) with θ = { θ 1 , θ 2 } , we have:
Revised: 1 ( Y l | A latent , G obs ) p θ 2 ( A latent |G obs ) with θ = { θ 1 , θ 2 } , we have:

Original: [Equation]
Revised: [Equation]

Original: Here, p θ 1 ( Y l | A latent , G obs ) in the ﬁrst term can be parameterized by arbitrary GCN models described by Eq. 1 that infer the node labels from A latent and X .
Revised: Here, p θ 1 ( Y l | A latent , G obs ) in the ﬁrst term can be parameterized by arbitrary GCN models described by Eq. 1 that infer the node labels from A latent and X .

Original: We use the vanilla GCN (Kipf & Welling, 2017) in this work (see Eq. 13 in Appendix A).
Revised: We use the vanilla GCN (Kipf & Welling, 2017) in this work (see Eq. 13 in Appendix A).

Original: The second term is the KL-divergence between q φ ( A latent |G obs ) and the prior p θ 2 ( A latent |G obs ) , which can be optimized by setting θ 2 = φ to force
Revised: The second term is the KL-divergence between q φ ( A latent |G obs ) and the prior p θ 2 ( A latent |G obs ) , which can be optimized by setting θ 2 = φ to force

Original: KL[ q φ ( A latent |G obs ) (cid:107) p θ 

Revised: KL[ q φ ( A latent |G obs ) (cid:107) p θ

Original: A latent |G obs )]
Revised: A latent |G obs )]

Original: = 0 .
Revised: = 0 .

Original: Actually, p θ 2 ( A latent |G obs ) is of little interest to the ﬁnal classiﬁcation task and we just need to maximize E q φ ( A latent | X ) [log p θ 1 ( Y l
Revised: Actually, p θ 2 ( A latent |G obs ) is of little interest to the ﬁnal classiﬁcation task and we just need to maximize E q φ ( A latent | X ) [log p θ 1 ( Y l

Original: |
Revised: |

Original: A latent , X )] in the M-step.
Revised: A latent , X )] in the M-step.

Original: Considering the fact that the observed graph structure A obs should not be fully discarded and the approximation q φ ( A latent |G obs ) derived in the previous E-step is sometimes not very accurate, we use q φ ( A latent |G obs ) to reﬁne A obs , and substitute q φ with the following ¯ q φ in practice:
Revised: Considering the fact that the observed graph structure A obs should not be fully discarded and the approximation q φ ( A latent |G obs ) derived in the previous E-step is sometimes not very accurate, we use q φ ( A latent |G obs ) to reﬁne A obs , and substitute q φ with the following ¯ q φ in practice:

Original: [Equation]
Revised: [Equation]

Original: where p ∈ (0 , 1] , ε 1 is close to one (commonly 0 . 999 ), and ε 2 is close to zero (commonly 0 . 01 ).
Revised: where p ∈ (0 , 1] , ε 1 is close to one (commonly 0 . 999 ), and ε 2 is close to zero (commonly 0 . 01 ).

Original: Eq.
Revised: Eq.

Original: 9 implies that, for edges predicted by q φ to be linked with high conﬁdence (the value after sigmoid or the maximum value after softmax), they should be added to the observed graph with probability p .
Revised: 9 implies that, for edges predicted by q φ to be linked with high conﬁdence (the value after sigmoid or the maximum value after softmax), they should be added to the observed graph with probability p .

Original: Edges predicted by q φ to be cut off with high conﬁdence should be removed from the observed graph.
Revised: Edges predicted by q φ to be cut off with high conﬁdence should be removed from the observed graph.

Original: Otherwise, we maintain the original graph structure with probability p .
Revised: Otherwise, we maintain the original graph structure with probability p .

Original: Similar to the E-step, we can sample the latent adjacency matrix ˆ A latent ∼ ¯
Revised: Similar to the E-step, we can sample the latent adjacency matrix ˆ A latent ∼ ¯

Original: q φ ( A latent |G obs ) (note that we pre-train p θ 1 using A obs ) and leverage SGD to minimize the cross-entropy error between the
Revised: q φ ( A latent |G obs ) (note that we pre-train p θ 1 using A obs ) and leverage SGD to minimize the cross-entropy error between the

Original: GCN’s predictions p θ 1 ( Y l | ˆ A latent , G obs )] and the ground-truth labels Y l for optimizing θ :
Revised: GCN’s predictions p θ 1 ( Y l | ˆ A latent , G obs )] and the ground-truth labels Y l for optimizing θ :

Original: [Equation]
Revised: [Equation]

Original: In the test procedure, the ﬁnal predictions for Y u are E ¯ q
Revised: In the test procedure, the ﬁnal predictions for Y u are E ¯ q

Original: [Equation]
Revised: [Equation]

Original: latent
Revised: latent

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Y u |
Revised: Y u |

Original: [Equation]
Revised: [Equation]

Original: , G obs )] , which can be approximated by Monte-Carlo sampling:
Revised: , G obs )] , which can be approximated by Monte-Carlo sampling:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the number of samples S and the probability p in ¯ q φ are tuned hyperparameters.
Revised: where the number of samples S and the probability p in ¯ q φ are tuned hyperparameters.

Original: The two neural networks q φ and p θ are trained in an alternating fashion to reinforce each other.
Revised: The two neural networks q φ and p θ are trained in an alternating fashion to reinforce each other.

Original: Topology optimization
Revised: Topology optimization

Original: in the E-step improves the performance of the GCN in the M-step, and with more unlabeled nodes being correctly classiﬁed, q φ will better approximate the optimal graph ˜ A .
Revised: in the E-step improves the performance of the GCN in the M-step, and with more unlabeled nodes being correctly classiﬁed, q φ will better approximate the optimal graph ˜ A .

Original: 3. D ISCUSSIONS
Revised: 3. D ISCUSSIONS

Original: In this subsection, we discuss the relationship between VEM-GCN and two recent works for tackling over-smoothing, i.e., DropEdge (Rong et al., 2020) and AdaEdge (Chen et al., 2020a).
Revised: In this subsection, we discuss the relationship between VEM-GCN and two recent works for tackling over-smoothing, i.e., DropEdge (Rong et al., 2020) and AdaEdge (Chen et al., 2020a).

Original: We show that these two methods are speciﬁc cases of VEM-GCN under certain conditions.
Revised: We show that these two methods are speciﬁc cases of VEM-GCN under certain conditions.

Original: More detailed comparisons with other related works (e.g., SBM-related GCNs) are discussed in Appendix B.
Revised: More detailed comparisons with other related works (e.g., SBM-related GCNs) are discussed in Appendix B.

Original: VEM-GCN vs. DropEdge.
Revised: VEM-GCN vs. DropEdge.

Original: DropEdge randomly removes a certain fraction of edges in each training step.
Revised: DropEdge randomly removes a certain fraction of edges in each training step.

Original: The authors proved that this strategy can retard the convergence speed of over-smoothing.
Revised: The authors proved that this strategy can retard the convergence speed of over-smoothing.

Original: However, it does not address the over-smoothing issue at the core, since the graph topology is not fundamentally optimized and noisy messages still pass along inter-class edges.
Revised: However, it does not address the over-smoothing issue at the core, since the graph topology is not fundamentally optimized and noisy messages still pass along inter-class edges.

Original: Considering the scenario where a node has few interactions with its community but many cross-community links,
Revised: Considering the scenario where a node has few interactions with its community but many cross-community links,

Original: DropEdge cannot improve the discrimination of this stray node, since it does not recover the missing intra-class edges.
Revised: DropEdge cannot improve the discrimination of this stray node, since it does not recover the missing intra-class edges.

Original: We ﬁnd that VEM-GCN degenerates to DropEdge, if we skip the E-step and just maximize E ¯ q φ ( A latent |G obs ) [log p θ 1 ( Y l | A latent , G obs )] with ¯ q φ ( a latent ij = 1 |G obs ) = p · a obs ij .[Table]
Revised: We ﬁnd that VEM-GCN degenerates to DropEdge, if we skip the E-step and just maximize E ¯ q φ ( A latent |G obs ) [log p θ 1 ( Y l | A latent , G obs )] with ¯ q φ ( a latent ij = 1 |G obs ) = p · a obs ij .

Original: VEM-GCN vs. AdaEdge.
Revised: VEM-GCN vs. AdaEdge.

Original: AdaEdge also constantly adjusts the graph topology in the training procedure.
Revised: AdaEdge also constantly adjusts the graph topology in the training procedure.

Original: It adds the edge between two nodes which are predicted by the GCN as the same class with high conﬁdence, and removes edges in a similar manner.
Revised: It adds the edge between two nodes which are predicted by the GCN as the same class with high conﬁdence, and removes edges in a similar manner.

Original: If we skip the E-step and set ¯ q φ as
Revised: If we skip the E-step and set ¯ q φ as

Original: Eq.
Revised: Eq.

Original: 12, VEM-GCN and AdaEdge can be equivalent.
Revised: 12, VEM-GCN and AdaEdge can be equivalent.

Original: [Equation]

Original: if y (cid:48) i = y (cid:48)

Original: [Equation]

Original: and conf( y (cid:48)

Original: [Equation]

Original: ) , conf( y (cid:48)

Original: [Equation]

Original: if y (cid:48)

Original: i (cid:54)

Original: [Equation]

Original: y (cid:48)

Original: [Equation]

Original: and conf( y (cid:48) i ) , conf( y (cid:48) j ) > τa obs ij , otherwise

Original: [Equation]
Revised: [Equation]

Original: 12) where y (cid:48) is the prediction made by GCN, conf( · ) denotes the corresponding conﬁdence, τ 1 and τare two thresholds.
Revised: where y (cid:48) is the prediction made by GCN, conf( · ) denotes the corresponding conﬁdence, τ 1 and τare two thresholds.

Original: Eq.
Revised: Eq.

Original: 12 implies that, this self-training-like fashion only adjusts the edges whose interacting nodes have already been classiﬁed with high conﬁdence.
Revised: 12 implies that, this self-training-like fashion only adjusts the edges whose interacting nodes have already been classiﬁed with high conﬁdence.

Original: Therefore, the performance improvement is limited and would even get worse for some misclassiﬁed nodes, as it might wrongly add inter-class edges to the observed graph A obs and remove helpful intra-class connections.
Revised: Therefore, the performance improvement is limited and would even get worse for some misclassiﬁed nodes, as it might wrongly add inter-class edges to the observed graph A obs and remove helpful intra-class connections.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To evaluate our VEM-GCN architecture, we conduct extensive experiments on seven benchmark datasets.
Revised: To evaluate our VEM-GCN architecture, we conduct extensive experiments on seven benchmark datasets.

Original: Underboth the sameexperimental setting as DropEdge (Rong et al., 2020) and a labelscarce setting (i.e., low label rate), we compare the performance of VEM-GCN against a variety of state of the arts for tackling over-smoothing, uncertain graphs and topology optimization in GCNs.
Revised: Under the same setting as DropEdge (Rong et al., 2020) and a label-scarce setting (i.e., low label rate), we compare the performance of VEM-GCN against a variety of state of the arts for tackling over-smoothing, uncertain graphs and topology optimization in GCNs.

Original: We further give the visualization results of topology optimization and quantitative analysis to verify the effectiveness of VEM-GCN in relieving the over-smoothing issue.
Revised: We further give the visualization results of topology optimization and quantitative analysis to verify the effectiveness of VEM-GCN in relieving the over-smoothing issue (complexity analysis is provided in Appendix E.3).

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: Datasets and Baselines.
Revised: Datasets and Baselines.

Original: We utilize seven well-known benchmark datasets to validate our proposed method.
Revised: We adopt seven well-known benchmark datasets to validate the proposed method.

Original: Cora (Sen et al., 2008), Cora-ML (McCallum et al., 2000; Bojchevski & G¨unnemann, 2018), Citeseer (Sen et al., 2008), and Pubmed (Namata et al., 2012) are four citation network benchmarks, where nodes represent documents and edges are citations between them.
Revised: Cora (Sen et al., 2008), Cora-ML (McCallum et al., 2000; Bojchevski & G ¨ unnemann, 2018), Citeseer (Sen et al., 2008), and Pubmed (Namata et al., 2012) are four citation network benchmarks, where nodes represent documents and edges are citations between documents.

Original: Amazon
Revised: Amazon

Original: Photo and Amazon Computers are two segments from the Amazon co-purchase graph (McAuley et al., 2015), in which nodes represent goods and edges indicate that two goods are frequently bought together.
Revised: Photo and Amazon Computers are two segments from the Amazon co-purchase graph (McAuley et al., 2015), in which nodes represent goods and edges indicate that two goods are frequently bought together.

Original: In the Microsoft Academic graph (Shchur et al., 2018), nodes are authors and edges represent their co-authorship.
Revised: In the Microsoft Academic graph (Shchur et al., 2018), nodes are authors and edges represent their co-authorship.

Original: All graphs usea bag-of-words encoded representation as node features.
Revised: All graphs use bag-of-words encoded representations as node attributes.

Original: An overview of the dataset statistics is summarized in Appendix C.
Revised: An overview of the dataset statistics is summarized in Appendix C.

Original: Since VEM-GCN aims at addressing the over-smoothing problem with graph topology optimization, we evaluate the node classiﬁcation performance of our method against various strategies for  tacklingover-smoothing, uncertain graphs and topology optimization in GCNs.
Revised: Since VEM-GCN aims at addressing the over-smoothing problem with graph topology optimization, we evaluate the node classiﬁcation performance of our method against various strategies for addressing the over-smoothing issue, tackling uncertain graphs and topology optimization in GCNs.

Original: For addressing the oversmoothing issue, four recently proposed methods are considered: DropEdge (Rong et al., 2020),[Table] AdaEdge (Chen et al., 2020a), PairNorm (Zhao & Akoglu, 2020) [Table]
Revised: For addressing the over-smoothing issue, four recent methods are considered: DropEdge (Rong et al., 2020), AdaEdge (Chen et al., 2020a), PairNorm (Zhao & Akoglu, 2020), and BBGDC (Hasanzadeh et al., 2020).

Original: and BBGDC (Hasanzadeh et al., 2020).

Original: For tackling uncertain graphs, we compare against several Bayesian approaches including BGCN (Zhang et al., 2019), VGCN (Tiao et al., 2019) [Table] and G 3 NN (Ma et al., 2019).
Revised: For tackling uncertain graphs, we compare against several Bayesian approaches including BGCN (Zhang et al., 2019), VGCN (Tiao et al., 2019), and G 3 NN (Ma et al., 2019).

Original: For topology optimization, LDS (Franceschi et al., 2019)
Revised: For topology optimization, LDS (Franceschi et al., 2019)

Original: , GDC (Klicpera et al., 2019b), and TO-GCN (Yang et al., 2019) are the baselines.
Revised: , GDC (Klicpera et al., 2019b), and TO-GCN (Yang et al., 2019) are the baselines.

Original: Another recent work termed GMNN (Qu et al., 2019), which also employed the variational EM algorithm for node classiﬁcation, is considered as a baseline as well.
Revised: GMNN (Qu et al., 2019) is also taken as a baseline, as it also employs the variational EM algorithm for node classiﬁcation.

Original: We conduct node classiﬁcation under two experimental  settings.
Revised: We conduct node classiﬁcation under two experimental settings, i.e., full-supervised and label-scarce settings.

Original: The ﬁrst one following DropEdge (Rong et al., 2020) is a full-supervised setting, where each dataset is split into 500 nodes for validation, 1000 nodes for test and the rest for training.
Revised: The full-supervised setting follows DropEdge (Rong et al., 2020), where each dataset is split into 500 nodes for validation, 1000 nodes for test and the rest for training.

Original: Thesecond one is a label-scarce setting, where only a few nodes are given labels, 500 nodesare selected for validation and the rest are set for test.
Revised: The label-scarce setting assigns labels to only a few nodes and selects 500 nodes for validation, while the rest are used for test.

Original: Under the label-scarce setting, we compare VEM-GCN with theabove baselines except for LDS, as it always uses the validation set for training, which is unfair for learning with limited training samples.
Revised: Under the label-scarce setting, we compare VEM-GCN with the baselines except for LDS, as LDS always uses the validation set for training, which is unfair for learning with limited training samples.

Original: Furthermore, considering that the classiﬁcation performance is highly inﬂuenced by the split of the dataset,for each evaluation we run all the models with the same 5 random data splits.
Revised: Considering that the classiﬁcation performance is highly inﬂuenced by the split of the dataset, we run all the models with the same 5 random data splits for each evaluation.

Original: To further ensure the credibility of the results, we perform 10 random weight initializations in each data split and report the average test accuracy for boththe two experimental settings.
Revised: To further ensure the credibility of the results, we perform 10 random weight initializations for each data split and report the average test accuracy for both experimental settings.

Original: Model Conﬁgurations.
Revised: Model Conﬁgurations.

Original: For a fairer comparison, we evaluate all the methods under the same GCN backbone and the same training procedure.
Revised: For fair comparison, we evaluate all the methods under the same GCN backbone and the same training procedure.

Revised: To be concrete, the graph model used in all baselines

Revised: [Table]

Revised: metrics, the larger the better.

Revised: A: Accuracy.

Original: To be concrete, the graph model used in all baselines and our VEM-GCN ( p θ  in the M-step) is a vanilla GCN (Kipf & Welling, 2017) with the number of hidden unitsbeing set as 32 .
Revised: S: Over-smoothness. [Table] and our VEM-GCN ( p θ 1 in the M-step) is a vanilla GCN (Kipf & Welling, 2017) with the number of hidden units set as 32 .

Original: Besides, we train all the models  with the same dropout rate of 0 .
Revised: Besides [Table] we train all the models for each dataset with the same dropout rate of 0 .

Original: 5 , the same weight decay, the same learning rate of 0 .
Revised: 5 , the same weight decay, the same learning rate of 0 .

Original: 01 , the same optimizer (Adam (Kingma &
Revised: 01 , the same optimizer (Adam (Kingma &

Original: Ba, 2015)),  and the same early stopping strategy based on the validation loss with a patience ofepochs. The maximum training epochs are manually tuned for each dataset.
Revised: Ba, 2015)), the same maximum training epoch of [Table] and the same early stopping strategy based on the validation loss with a patience of 50 epochs.

Original: For q φ in the E-step, the input node embeddings are the attributes averaged overeach node’s neighborhoods and the structure of  theneural network  is a four-layer MLP with hidden units of size 128 , 64 , 64 , and 32 , respectively.
Revised: For q φ in the E-step, the input node embeddings are the attributes averaged over the neighborhood of each node and the network architecture is a four-layer MLP with hidden units of size 128 , 64 , 64 , and 32 , respectively.

Original: More details of the hyperparameter settings for each dataset are listed in Appendix D.
Revised: Please refer to Appendix D for more details of the hyperparameter settings for each dataset.

Original: 4. R ESULTS AND A NALYSIS
Revised: 4. R ESULTS AND A NALYSIS

Original: Full-supervised Setting.
Revised: Full-supervised Setting.

Original: Table 1 summarizes the classiﬁcation results.
Revised: Table 1 summarizes the classiﬁcation results.

Original: The highest accuracy in each column is highlighted in bold.
Revised: The highest accuracy in each column is highlighted in bold.

Original: Note that the results of BBGDC and LDS on three large graphs (i.e., Pubmed, Amazon Computers, and MS Academic) is missing due to the out-of-memory error.
Revised: Note that the results of BBGDC and LDS on three large graphs (i.e., Pubmed, Amazon Computers, and MS Academic) are missing due to the out-of-memory error.

Original: The results demonstrate that none of the baselines outperform the vanilla GCN in all cases, while VEM-GCN consistently improves the test accuracy of the GCN backbone by signiﬁcant margins.
Revised: Table 1 demonstrates that none of the baselines outperform the vanilla GCN in all cases, while VEMGCN consistently improves the test accuracy of the GCN backbone by noticeable margins.

Original: Speciﬁcally, we make the following observations under the full-supervised setting: (1) For tackling over-smoothing, AdaEdge, DropEdge and PairNorm demonstrate limited improvement on several datasets, while BBGDC almost collapses for all cases.
Revised: Speciﬁcally, we ﬁnd the following facts under the full-supervised setting: (1) For tackling over-smoothing, AdaEdge, DropEdge and PairNorm demonstrate limited improvement on several datasets, while BBGDC almost collapses for all cases.

Original: (2)For LDS, TO-GCN, and GDC, there is no guarantee that their topology optimization could achieve performance gains for the GCN backbone.
Revised: (2) LDS, TO-GCN, and GDC cannot guarantee that their topology optimization could achieve performance gains for the GCN backbone.

Original: (3)For Bayesian approaches and GMNN, only  comparable performance with the vanilla GCNare achieved in almost all cases.
Revised: (3) Bayesian approaches and GMNN can only achieve comparable performance with the vanilla GCN in almost all cases.

Original: Overall, these facts imply that VEM-GCN signiﬁcantly beneﬁts from the large labeled data to generate a clearer topology that leads to better performance.
Revised: Overall, these facts imply that VEM-GCN signiﬁcantly beneﬁts from the large labeled data to generate a clearer topology and achieve better performance.

Original: Label-scarce Setting.
Revised: Label-scarce Setting.

Original: We randomly select 10 labeled nodes per class as the training set and evaluate the performance of VEM-GCN with varying layers.
Revised: We randomly select 10 labeled nodes per class as the training set and evaluate the performance of VEM-GCN with varying layers.

Original: Table 2 shows the test accuracy and the over-smoothness measurements of the learned node embeddings (input node features of the last 
Revised: Table 2 shows the test accuracy and the oversmoothness measurements of the learned node embeddings (input node features of the last layer).

Original: [Table]
Revised: [Table]

Original:  indeed produces more seperable across 
Revised: The metric to measure the over-smoothness is deﬁned in Appendix D.3 and supplementary results on additional datasets are shown in Appendix E.1. The vanilla GCN severely suffers from the oversmoothing issue, while VEM-GCN can achieve performance gains even with deeper layers. The results for over-smoothness measurement indicate that VEM-GCN indeed produces more separable node embeddings across different classes.

Original: We further take Amazon Photo as an example dataset to validate VEM-GCN under different label rates.
Revised: We further take Amazon Photo as an example dataset to validate VEM-GCN under different label rates.

Original: Similar trend as Table 1 can be seen in Table 3.
Revised: Similar trend as Table 1 can be found in Table 3.

Original: Convergence Analysis and Visualization Results.
Revised: Visualization Results.

Original: VEM-GCN utilizes the variational EM algorithm for optimization.
Revised: VEM-GCN leverages the variational EM algorithm for optimization.

Original: In this subsection, we analyze the convergence of VEM-GCN.
Revised: In this subsection, we analyze the convergence of VEM-GCN.

Original: Figuredepicts the accuracy improvement curve of p θ  during the EM iterations under the full-supervised setting.
Revised: Figuredepicts the accuracy improvement curve of p θ 1 during the EM iterations under the full-supervised setting.

Original: We can observe that the accuracy consistently increases during the EM iterations.
Revised: We ﬁnd that VEM-GCN requires only a few iterations to converge.

Original: We further take Cora-ML as an example to give the corresponding visualization results of graph topology optimization in Figure 2.
Revised: We further take CoraML as an example to give the corresponding visualization results of graph topology optimization.

Original: As can be seen, the observed graph is very sparse and contains a few intraclass edges, while the optimized graph recover many missing intra-class edges to relieve the oversmoothing problem.
Revised: Figure 2 show that the observed graph is very sparse and contains a few intra-class edges, while the optimized graph recover many missing intra-class edges to relieve the over-smoothing problem.

Original: Note that although the reﬁned graph is much denser than the observed graph, the parameter p ( 0 [-. 
 here)-] in ¯ q φ helps maintain the sparsityproperty of the adjacency matrix, which makes the M-step can still be implemented efﬁciently using sparse-dense matrix multiplications.
Revised: Note that, although the reﬁned graph is much denser than the observed graph, the parameter p ( 0 [-. 
 here)-].here) in ¯ q φ helps maintain the sparsity of the adjacency matrix in the training procedure. Thus, the M-step can still be implemented efﬁciently using sparse-dense matrix multiplications.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present a novel architecture termed VEM-GCN for addressing the over-smoothing problem in GCNs with graph topology optimization.
Revised: In this paper, we present a novel architecture termed VEM-GCN for addressing the over-smoothing problem in GCNs with graph topology optimization.

Original: By introducing a latent graph parameterized by the assortative-constrained stochastic block model and utilizing the variational EM algorithm to jointly optimize the graph structure and the likelihood function, VEM-GCN outperforms a variety of state-of-the-art methods for tackling over-smoothing, uncertain graphs, and topology optimization in GCNs.
Revised: By introducing a latent graph parameterized by the assortative-constrained stochastic block model and utilizing the variational EM algorithm to jointly optimize the graph structure and the likelihood function, VEM-GCN outperforms a variety of state-of-the-art methods for tackling over-smoothing, uncertain graphs, and topology optimization in GCNs.

Original: For future work, we expect further improvements for the VEM-GCN architecture to deal with more complex graphs such as hypergraphs and heterogeneous graphs.
Revised: For future work, we expect further improvements for the VEM-GCN architecture to deal with more complex graphs such as hypergraphs and heterogeneous graphs.

Original: A A LGORITHM
Revised: A A LGORITHM

Original: For a fair comparison, we adopt the vanilla GCN (Kipf & Welling, 2017) as the backbone for the proposed VEM-GCN architecture and all the baselines.
Revised: For a fair comparison, we adopt the vanilla GCN (Kipf & Welling, 2017) as the backbone for all baselines and our proposed VEM-GCN architecture.

Original: A two-layer GCN is in the following form:
Revised: A two-layer GCN is in the following form:

Original: [Equation]
Revised: [Equation]

Original: where X is the input node attribute matrix, I N is the identity matrix, ˙
Revised: where X is the input node attribute matrix, I N is the identity matrix, ˙

Original: A =
Revised: A =

Original: A + I N is the adjacency matrix with added self-loops, ˙ D is its corresponding diagonal degree matrix, and θ  = { Θ ( 0 ) , Θ ( 1 ) } are the learnable weight parameters.
Revised: A + I N is the adjacency matrix with added self-loops, ˙ D is its corresponding diagonal degree matrix, and θ 1 = { Θ ( 0 ) , Θ ( 1 ) } are the learnable weight parameters.

Original: Algorithm 1 elaborates the proposed VEM-GCN architecture.
Revised: Algorithm 1 elaborates the proposed VEM-GCN architecture.

Original: B F URTHER D ISCUSSIONS
Revised: B F URTHER D ISCUSSIONS

Revised: In addition to recent strategies for tackling over-smoothing issues, we further distinguish VEM-GCN from the SBM-related GCNs and VGCN and GMNN (that introduce variational inference).

Original: Comparison withOther SBM-related GCNs.
Revised: Comparison with SBM-related GCNs.

Original: Stochastic block model (SBM) has been combined with GCNs in several recent works (i.e., BGCN (Zhang et al., 2019) and G 3 NN (Ma et al., 2019)).
Revised: Stochastic block model (SBM) has been combined with GCNs in several recent works (i.e., BGCN (Zhang et al., 2019) and G 3 NN (Ma et al., 2019)).

Original: However,the motivations, objective functions and training methods of these architectures are totally different from that of VEM-GCN.
Revised: However, these architectures are totally different from VEM-GCN in motivations, objective functions and training methods.

Original: BGCN (Zhang et al., 2019) jointly infers the node labels and the parameters of SBM using only A obs , which ignores the dependence of the graph on X and Y l .
Revised: BGCN (Zhang et al., 2019) jointly infers the node labels and the parameters of SBM using only A obs , which ignores the dependence of the graph on X and Y l .

Original: Then theGCN is trained with adjacency matrices sampled from the inferred SBM.
Revised: Then the adjacency matrices sampled from the inferred SBM are used to train the GCN.

Original: Different from VEMGCN, BGCN does not explicitly promote intra-class connection and demote inter-class interaction.
Revised: Different from VEMGCN, BGCN neither explicitly promotes intra-class connection nor demotes inter-class interaction.

Original: It only achieves robustness under certain conditions such as adversarial attacks, beneﬁting from the uncertainty brought by the inferred SBM.
Revised: It only achieves robustness under certain conditions such as adversarial attacks, beneﬁting from the uncertainty brought by the inferred SBM.

Original: G 3 NN is a ﬂexible generative model, where the graph generated by SBM is based on the predictions of an additional MLP learned from only X and Y l .
Revised: G 3 NN is a ﬂexible generative model, where the graph generated by SBM is based on the predictions of an additional MLP learned from only X and Y l .

Original: The predictions for the unlabeled nodes are still based on G obs (i.e., the input adjacency matrix of the GCN is still A obs ).
Revised: The predictions for the unlabeled nodes are still based on G obs (i.e., the input adjacency matrix of the GCN is still A obs ).

Original: In comparison, VEM-GCN is aimed at addressing the over-smoothing issue with topology optimization.
Revised: By contrast, VEM-GCN aims at addressing the over-smoothing issue with topology optimization.

Original: In VEM-GCN, the M-step trains a GCN to obtain the predictions of the unlabeled nodes based on A latent , X , and Y l .
Revised: In VEM-GCN, the M-step trains a GCN to obtain the predictions of the unlabeled nodes based on A latent , X , and Y l .

Original: We then estimate the posterior distribution on A latent based on Y l and the predictions for the unlabeled nodes with the SBM assumption, as we ﬁnd that theoptimal graph can be modeled as a SBM with p 0 = 1 and p 1 = 0 in Eq.
Revised: We then estimate the posterior distribution on A latent based on Y l and the predictions for the unlabeled nodes under the SBM assumption.

Original: Subsequently, the E-step optimizes the graph topology by training another auxiliary neural network with node embeddings as input to approximate the posterior distribution.
Revised: Subsequently, the E-step optimizes the graph topology by training another auxiliary neural network with node embeddings as input to approximate the posterior distribution of A latent .

Original: The E-step and M-step are optimized in an alternating fashion to reinforce each other.
Revised: The E-step and M-step are optimized in an alternating fashion to improve each other.

Original: VEM-GCN vs. VGCN.
Revised: VEM-GCN vs. VGCN.

Original: Tiao et al. (2019) proposed VGCN, which also introduces a latent graph A latent and optimizes L ELBO in Eq.
Revised: VGCN (Tiao et al., 2019) also introduces a latent graph A latent and optimizes L ELBO in Eq.

Original: However, it directly optimizes the ELBO in a VAE (Kingma & Welling, 2014) fashion and the posterior distribution of A latent is set to approximate the predeﬁned graph priors p ( a prior ij = 1)
Revised: However, it directly optimizes the ELBO in a VAE (Kingma & Welling, 2014) fashion and the posterior distribution of A latent is set to approximate the pre-deﬁned graph priors p ( a prior ij = 1)

Original: =
Revised: =

Original: ρ 1 a obs ij + ρ 2 (1 − a obs ij ) with 0 < ρ 1 , ρ 2 < 1 using the reparameterization trick.
Revised: ρ 1 a obs ij + ρ 2 (1 − a obs ij ) with 0 < ρ 1 , ρ 2 < 1 using the re-parameterization trick.

Original: Their method is aimed at achieving robustness under fake link attacks and only shows comparable performance with GCN under the standard transductive learning setting that infers Y u based on G obs .
Revised: VGCN is to achieve robustness under fake link attacks and only shows comparable performance with GCN under the standard transductive learning setting (i.e., inferring Y u based on G obs ).

Original: In comparison, VEM-GCN does not introduce priors over graphs.
Revised: By contrast, VEM-GCN does not introduce priors over graphs.

Original: We optimize the  topology by explicitly promoting intra-class connection and demoting inter-class interaction using SBM and variational EM to relieve the over-smoothing issue.
Revised: We optimize the graph topology by explicitly enhancing intra-class connection and suppressing inter-class interaction using SBM and variational EM to relieve the over-smoothing issue.

Original: VEM-GCN vs. GMNN.
Revised: VEM-GCN vs. GMNN.

Original: Variational EM was also employed for node classiﬁcation in the recent work termed Graph Markov Neural Network (GMNN) (Qu et al., 2019), but GMNN aims at modeling the graph as a conditional random ﬁeld and is totally different from our method in motivations andthe objective functions. 
Revised: Graph Markov Neural Network (GMNN) (Qu et al., 2019) also employs variational EM for node classiﬁcation, but it is totally different from our method in motivations and objective functions. GMNN focuses on modeling the joint distribution of object (node) labels. Therefore, GMNN views Y u as latent variables and optimizes the following ELBO:

Original: Pros and Cons.
Revised: [Equation]

Original: Our VEM-GCN is very ﬂexible and general. There is no constraint on the design of the two neural networks q φ  and  p θ  , making the architecture able to be combined with arbitrary GCN models and node embedding methods. Moreover, our method generalizes some existing state-of-
Revised: In the E-step, GMNN parameterizes q φ ( Y u | X ) with a GCN and q φ ( Y u | X ) is optimized to approximate the posterior distribution p θ ( Y u | Y l , X ) . In the M-step, GMNN utilizes another GCN to model the conditional distribution p θ ( y i | y NB ( i ) , X ) for each node v i ∈ V (NB ( i ) is the neighbor

Original: [Table]
Revised: [Table]

Original: the-art strategies for tackling the over-smoothing problem (i.e., DropEdge and AdaEdge). Besides, VEM-GCN also has some disadvantages, because the performance improvement brought by topology optimization is at the cost of efﬁciency. As illustrated in Sections 3.2 and 3.3, the M-step is a traditional training procedure for optimizing the GCNs, while the E-step introduces an extra procedure that trains a MLP for graph structure optimization.
Revised: set of node v i ) with a conditional random ﬁeld and maximizes the corresponding likelihood.

Original: However, to address the over-smoothing issue at the core, we argue that graph topology optimization is necessary. Actually, efﬁciency issue is a common problem for some Bayesian approaches and topology optimization methods. Despite losing efﬁciency compared with the vanilla GCN, VEM-GCNoptimizes A latent with amortized variational inference (i.e., training a MLP shared by all the node pairs with mini-batch SGD in the E-step), which is faster than BGCN (Zhang et al., 2019)
Revised: On the contrary, VEM-GCN is proposed to relieve the over-smoothing issue.

Original: (Bayesian method) and LDS (Franceschi et al., 2019) (topology optimization) and is scalable for large-scale graphs. For training on  the Amazon Photo dataset with a GTX 1080Ti GPU, VEM-GCN is about 3x faster than LDS and 4x faster than BGCN.
Revised: VEM-GCN optimizes the latent graph to approximate its posterior distribution based on SBM in the E-step, and trains a GCN based on the latent graph in the M-step.

Original: C D ATASET S TATISTICS
Revised: C D ATASET S TATISTICS

Original: We utilize seven node classiﬁcation benchmarks in this paper, including four citation networks (i.e.,
Revised: We utilize seven node classiﬁcation benchmarks in this paper, including four citation networks (i.e.,

Original: Citeseer, Pubmed, Cora, and Cora -ML ), two Amazon co purchase graphs i.e., Amazon Photo and
Revised: Citeseer, Pubmed, Cora, and Cora-ML), two Amazon co-purchase graphs (i.e., Amazon Photo and

Original: Amazon Computers), and one Microsoft Academic graph, as summarized below.
Revised: Amazon Computers), and one Microsoft Academic graph, as summarized below.

Original: • Citation Networks .
Revised: • Citation Networks .

Original: Cora, Citeseer, Pubmed can be downloaded from the ofﬁcial source code of GCN (Kipf & Welling, 2017) publicly available at https://github.com/ tkipf/gcn/tree/master/gcn/data , and Cora-ML can be downloaded from the source code of (A)PPNP (Klicpera et al., 2019a) publicly available at https:// github.com/klicperajo/ppnp/tree/master/ppnp/data .
Revised: Cora, Citeseer, Pubmed can be downloaded from the ofﬁcial source code of GCN (Kipf & Welling, 2017) publicly available at https://github.com/ tkipf/gcn/tree/master/gcn/data , and Cora-ML can be downloaded from the source code of (A)PPNP (Klicpera et al., 2019a) publicly available at https:// github.com/klicperajo/ppnp/tree/master/ppnp/data .

Original: • Amazon Co-purchase Graph .
Revised: • Amazon Co-purchase Graph .

Original: The Amazon Photo and Amazon Computers datasets from the Amazon co-purchase graph can be publicly downloaded from https://github.
Revised: The Amazon Photo and Amazon Computers datasets from the Amazon co-purchase graph can be publicly downloaded from https://github.

Original: com/shchur/gnn-benchmark/tree/master/data (Shchur et al., 2018).
Revised: com/shchur/gnn-benchmark/tree/master/data (Shchur et al., 2018).

Original: • Microsoft Academic Graph .
Revised: • Microsoft Academic Graph .

Original: The MS Academic graph can be downloaded from the source code of (A)PPNP (Klicpera et al., 2019a) publicly available at https://github.com/ klicperajo/ppnp/tree/master/ppnp/data .
Revised: The MS Academic graph can be downloaded from the source code of (A)PPNP (Klicpera et al., 2019a) publicly available at https://github.com/ klicperajo/ppnp/tree/master/ppnp/data .

Original: An overview of the dataset statistics is listed in Table 4.
Revised: An overview of the dataset statistics is listed in Table 4.

Original: Note that for these open datasets, three (Cora, Citeseer, Pubmed) are given in the form of undirected graphs, while four (Cora-ML, Amazon
Revised: Note that for these open datasets, three (Cora, Citeseer, Pubmed) are given in the form of undirected graphs, while four (Cora-ML, Amazon

Original: Photo, Amazon Computers, MS Academic) are directed graphs.
Revised: Photo, Amazon Computers, MS Academic) are directed graphs.

Original: GCN treats all these datasets as undirected graphs (i.e., a (cid:48) ij = [ a ij + a ji > 0] , where [ · ] denotes Iverson bracket).
Revised: GCN treats all these datasets as undirected graphs (i.e., a (cid:48) ij = [ a ij + a ji > 0] , where [ · ] denotes Iverson bracket).

Original: D F URTHER E XPERIMENTAL D ETAILS D. I MPLEMENTATIONS
Revised: D F URTHER E XPERIMENTAL D ETAILS D. I MPLEMENTATIONS

Original: The implementation of VEM-GCN consists of two alternating steps, including M-step and E-step.
Revised: The implementation of VEM-GCN consists of two alternating steps in each iteration, including a variational E-step and an M-step.

Original: In the M-step, p θ is a vanilla GCN with the number of hidden units set to 32 , and we use the ofﬁcial source code from https://github.com/tkipf/gcn/tree/master/gcn .

Original: In the  E-step, a simple four-layer MLP is implemented for q φ , where the numbers of neuron units of each layer are 128 , 64 , 64 , and 32 , respectively.
Revised: In the variational E-step, a simple four-layer MLP is implemented for q φ , where the numbers of neuron units of each layer are 128 , 64 , 64 , and 32 , respectively.

Original: We use tanh as the nonlinear activation function for the hidden layers.
Revised: We use tanh as the nonlinear activation function for the hidden layers.

Revised: In the M-step, p θ 1 is a vanilla GCN with the number of hidden units set as 32 , and we use the ofﬁcial source code from https://github.com/tkipf/gcn/tree/master/gcn .

Revised: All the baselines and our VEMGCN architecture are trained on a single NVIDIA GTX 1080 Ti GPU with 11GB memory.

Original: We just utilize the raw node attributes X as the input to q φ .
Revised: We just utilize the raw node attributes X as the input to q φ .

Original: Note that we can also support any other desirable network embedding method and these experiments is left for the future work.
Revised: Note that we can also support any other desirable network embedding method and these experiments is left for the future work.

Original: Considering the fact that the bag-of-words representation of X is often noisy, we average the features of each node over its neighborhoods to smooth the input signal.
Revised: Considering the fact that the bag-of-words representations of X is often noisy, we average the features of each node over its neighborhoods to smooth the input signal.

Original: Let ˆ A row =
Revised: Let ˆ A row =

Original: ( D + γ I N ) − 1 ( A + γ
Revised: ( D + γ I N ) − 1 ( A + γ

Original: I N ) denote the “self-enhanced” adjacency matrix with row normalization (we use γ = 1 . 5 in this paper), and [ A , B ]  the concatenation of  A and B , Consequently, we summarize the speciﬁc input to q φ for all the datasets as below.
Revised: I N ) denote the “self-enhanced” adjacency matrix with row normalization (we use γ = 1 . 5 in this paper), and [ A (cid:107) B ] be the concatenation of matrices A and B along the last dimension. Consequently, we summarize the speciﬁc input to q φ for all the datasets as below.

Original: • For Cora, Citeseer, and MS Academic, we use X (cid:48)
Revised: • For Cora, Citeseer, and MS Academic, we use X (cid:48)

Original: =
Revised: =

Original: ˆ A row X as the input of q φ .
Revised: ˆ A row X as the input of q φ .

Original: • For Pubmed and Amazon Photo, we use X (cid:48)
Revised: • For Pubmed and Amazon Photo, we use X (cid:48)

Original: = (cid:104) X , ˆ A row X , ˆ A 2 row X (cid:105) as the input of q φ .
Revised: = (cid:104) X (cid:107) ˆ A row X (cid:107) ˆ A 2 row X (cid:105) as the input of q φ .

Original: • For Cora-ML and Amazon Computers, we use X (cid:48)
Revised: • For Cora-ML and Amazon Computers, we use X (cid:48)

Original: = (cid:104) X ,
Revised: = (cid:104) X (cid:107)

Original: ˆ A row X (cid:105) as the input of q φ .
Revised: ˆ A row X (cid:105) as the input of q φ .

Original: For the sampling in the E-step, we ﬁnd that it is not always stable to draw the sampled labels ˆ
Revised: For the sampling in the E-step, we ﬁnd that it is not always stable to draw the sampled labels ˆ

Original: Y u ∼ p θ ( Y u | Y l , G obs ) .
Revised: Y u ∼ p θ ( Y u | Y l , G obs ) .

Original: To alleviate this problem, we maintain ˆ Y u = argmax y
Revised: To alleviate this problem, we maintain ˆ Y u ← argmax y

Original: ( p θ ( Y u | Y l , G obs )) withthe probability p e , and sample
Revised: ( p θ ( Y u | Y l , G obs )) with probability p e , and sample

Original: ˆ Y u ∼ p θ ( Y u | Y l , G obs ) withthe probability (1 − p e ) to improve the performance in practice.
Revised: ˆ Y u ∼ p θ ( Y u | Y l , G obs ) with probability (1 − p e ) to improve the performance in practice.

Original: In the test procedure, we perform inference in two ways: (1) keep p in Eq. 9 the same as the training procedure (commonly p < 1 ), sample S ( S > 1) adjacency matrices, and predict the classes for the unlabeled nodes according to Eq. 11; (2) set p = 1 (i.e., the latent adjacency matrix in now deterministic) and S = 1 .
Revised: In the test procedure, we perform inference in two ways: (1) keep p in Eq. 9 the same as the training procedure (commonly p < 1 ), sample S ( S > 1) adjacency matrices, and predict the classes for the unlabeled nodes according to Eq. 11; (2) set p = 1 (i.e., the latent adjacency matrix in now deterministic) and S = 1 .

Original: We report the best test accuracy obtained using these two sampling methods on each dataset and ﬁnd that (2) almost always performs better [Table]
Revised: We report the best test accuracy obtained using these two sampling methods on each dataset and ﬁnd that (2) almost always performs better.

Original: D. H YPERPARAMETER S ETTINGS
Revised: D. H YPERPARAMETER S ETTINGS

Original: Table 5 summarizes the hyperparameters adopted for the full-supervised setting on the seven node classiﬁcation benchmarks.
Revised: Table 5 summarizes the hyperparameters adopted for the full-supervised setting on the seven node classiﬁcation benchmarks.

Original: For the label-scarce setting, the hyperparameters are the same, except for ε 1 that needed to be carefully tuned in the search space { 0 . 99 , 0 . 995 , 0 . 999 , 0 .
Revised: For the label-scarce setting, the hyperparameters are the same, except for ε 1 that needed to be carefully tuned in the search space { 0 . 99 , 0 . 995 , 0 . 999 , 0 .

Original: 9995 , 0 . 9999 } for each data split.
Revised: 9995 , 0 . 9999 } for each data split.

Original: {+a obs ij = 0) = 
+}
Revised: {+a obs ij = 0) = 
+}

Revised: and λ ( a obs ij = 1)

Revised: =

Revised: λ .

Revised: lr: Learning rate.

Original: [Table]
Revised: [Table]

Original: D. M ETRIC FOR M EASURING O VER - SMOOTHNESS
Revised: D. M ETRIC FOR M EASURING O VER - SMOOTHNESS

Original: To address the over-smoothing issue, one would prefer to reduce the intra-class distance to make node features in the same class similar, and increase the inter-class distance to produce distinguishable representations for nodes in different classes.
Revised: To address the over-smoothing issue, one would prefer to reduce the intra-class distance to make node features in the same class similar, and increase the inter-class distance to produce distinguishable representations for nodes in different classes.

Original: Hence, we use the ratio of average inter-class distance to average intra-class distance (Euclidean distance of input node features in the last layer) to measure the over-smoothness.
Revised: Hence, we use the ratio of average inter-class distance to average intra-class distance (Euclidean distance of input node features in the last layer) to measure the over-smoothness.

Original: Formally, given the learned node embeddings H = { h i } Ni =1 (the input node features of the last layer), we ﬁrst calculate the distance matrix D =
Revised: Formally, given the learned node embeddings H = { h i } Ni =1 (the input node features of the last layer), we ﬁrst calculate the distance matrix D =

Original: [ d ij ] ∈ R N × N with each entry deﬁned as
Revised: [ d ij ] ∈ R N × N with each entry deﬁned as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where (cid:107) · (cid:107) 2 denotes
Revised: where (cid:107) · (cid:107) 2 denotes

Original: Euclidean norm.
Revised: Euclidean norm.

Original: Next, we deﬁne the inter-class mask matrix and intra-class mask matrix as follows:
Revised: Next, we deﬁne the inter-class mask matrix and intra-class mask matrix as follows:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  where ˜ A =
Revised: 17) where ˜ A =

Original: YY (cid:62) is the optimal graph and 1 N × N is a matrix of size N × N with all entriesbeing set as 1 .
Revised: YY (cid:62) is the optimal graph and 1 N × N is a matrix of size N × N with all entries set to 1 .

Original: Then we can obtain the inter-class distance matrix
Revised: Then we can obtain the inter-class distance matrix

Original: D inter =
Revised: D inter =

Original: [ d interij ] ∈ R N × N and the intra-class distance matrix D intra
Revised: [ d interij ] ∈ R N × N and the intraclass distance matrix D intra

Original: =
Revised: =

Original: [ d intraij ] ∈ R N × N by element-wise multiplication D with the mask matrices:
Revised: [ d intra ij ] ∈ R N × N by element-wise multiplication D with the mask matrices:

Original: [Equation]
Revised: [Equation]

Original: where ◦ denotes element-wise multiplication.
Revised: where ◦ denotes element-wise multiplication.

Original: Next we get the average inter-class distance AD inter and the average intra-class distance AD intra , with which we measure the over-smoothness as their ratio R :
Revised: Next we get the average inter-class distance AD inter and the average intra-class distance AD intra , with which we measure the over-smoothness as their ratio R :

Original: [Equation]
Revised: [Equation]

Original: where 1 ( x ) = 1 if x > 0 otherwise 0 .
Revised: where 1 ( x ) = 1 if x > 0 otherwise 0 .

Original: E F URTHER E XPERIMENTAL R ESULTS E. C LASSIFICATION R ESULTS UNDER THE L ABEL -S CARCE S ETTING
Revised: E F URTHER E XPERIMENTAL R ESULTS E. C LASSIFICATION R ESULTS UNDER THE L ABEL -S CARCE S ETTING

Original: Supplementary experiments for Table 2 are shown in Table6.
Revised: Supplementary experiments for Table 2 are shown in Table

Original: labels per class) with varying layers.

Original: For both metrics, the larger the better.

Original: A: Accuracy.

Original: S:

Original: Over-smoothness.

Original: [Table]
Revised: [Table]

Original: V ISUALIZATION R ESULTS
Revised: V ISUALIZATION R ESULTS

Original: Figure 2 demonstrates the topology optimization results on the Cora-ML dataset under the fullsupervised setting.
Revised: Figure 2 demonstrates the topology optimization results on the Cora-ML dataset under the fullsupervised setting.

Original: For the lable-scarce setting, the visualization results are shown in Figure 3.
Revised: For the lable-scarce setting, the visualization results are shown in Figure 3.

Original: We further take Cora-ML as an example dataset to provide t-SNE (Maaten & Hinton, 2008) visualizations of the learned node embeddings (input node features of the last layer) extracted by the vanilla GCN and our proposed VEM-GCN with varying layers under the label-scarce setting (labeled nodes per class).
Revised: We further take Cora-ML as an example dataset to provide t-SNE (Maaten & Hinton, 2008) visualizations of the learned node embeddings (input node features of the last layer) extracted by the vanilla GCN and our proposed VEM-GCN with varying layers under the label-scarce setting (labeled nodes per class).

Original: The results are shown in Figure 4 and 5.
Revised: The results are shown in Figures 4 and 5.

Original: As can be seen, VEM-GCN indeed generates more separable node embeddings for nodes in different classes (colors) for classiﬁcation.
Revised: As can be seen, VEM-GCN indeed generates more separable node embeddings for nodes in different classes (colors) for classiﬁcation.

Original: In particular, a ten-layer vanilla GCN severely suffers from the over-smoothing issue where node features in different classes are over-mixed and thus indistinguishable, while our VEM-GCN architecture with a ten-layer GCN as the backbone still achieves comparable performance with a two-layer model. 
Revised: In particular, a ten-layer vanilla GCN severely suffers from the over-smoothing issue where node features in different classes are over-mixed and thus indistinguishable, while our VEM-GCN architecture with a ten-layer GCN as the backbone still achieves comparable performance with a two-layer model. E. C OMPLEXITY A NALYSIS VEM-GCN is very ﬂexible and general.

Revised: There is no constraint on the design of the two neural networks q φ and p θ .

Revised: Therefore, the architecture can be combined with arbitrary GCN models and node

Revised: [Table]

Original: E. VEM-GCNII
Revised: embedding methods. Moreover, we generalize some existing state-of-the-art strategies for tackling the over-smoothing problem (i e., DropEdge and AdaEdge). In comparison to the vanilla GCN, VEM-GCN achieves these beneﬁts with topology optimization at the cost of efﬁciency.

Revised: As elaborated in Sections 3.2 and 3.3, the M-step is a traditional training procedure for optimizing the GCNs, while the E-step introduces an extra procedure that trains a MLP for graph structure optimization.

Revised: However, to address the over-smoothing issue at the core, we argue that graph topology optimization is necessary.

Revised: Actually, efﬁciency issue is a common problem for some Bayesian approaches and topology optimization methods.

Revised: Despite decreased efﬁciency compared with the vanilla GCN,

Revised: VEM-GCN optimizes A latent with amortized variational inference (i.e., training a MLP shared by all the node pairs with mini-batch SGD in the E-step), which is faster than BGCN (Zhang et al., 2019)

Revised: (Bayesian method) and LDS (Franceschi et al., 2019)

Revised: (topology optimization) and is scalable for large-scale graphs.

Revised: For training on the Amazon Photo dataset with a NVIDIA GTX 1080 Ti GPU,

Revised: VEM-GCN is about 3x faster than LDS and 4x faster than BGCN.

Revised: E. VEM-GCNII

Original: As mentioned above, our VEM-GCN architecture is ﬂexible and general.
Revised: As mentioned above, our VEM-GCN architecture is ﬂexible and general.

Original: In the E-step, the neural network can support arbitrary desirable node embeddings and the GCN in the M-step can be substituted with any graph models.
Revised: In the E-step, the neural network can support arbitrary desirable node embeddings and the GCN in the M-step can be substituted with any graph models.

Original: This subsection further veriﬁes the effectiveness of our proposed method by trying different models for p θ 1 ( Y l | A latent , G obs ) .
Revised: This subsection further veriﬁes the effectiveness of our proposed method by trying different models for p θ 1 ( Y l | A latent , G obs ) .

Original: We also apply the same node embeddings as illustrated in Appendix D. Trying more effective node embeddings is not the focus in this paper and is left for the future work.
Revised: We also apply the same node embeddings as illustrated in Appendix D. Trying more effective node embeddings is not the focus of this paper and is left for the future work.

Original: Recently, Chen et al.
Revised: Recently, Chen et al.

Original: (2020b) proposed a simple and deep GCN model termed GCNII to address the over-smoothing problem.
Revised: (2020b) proposed a simple and deep GCN model termed GCNII to address the over-smoothing problem.

Original: GCNII improves the vanilla GCN via Initial residual and Identity mapping:
Revised: GCNII improves the vanilla GCN via Initial residual and Identity mapping:

Original: [Equation]
Revised: [Equation]

Original:  where H (0) =
Revised: ) where H (0) =

Original: XW (0) is the output of the ﬁrst layer (a fully connected layer), ˜ P
Revised: XW (0) is the output of the ﬁrst layer (a fully connected layer), ˜ P

Original: = ˙
Revised: = ˙

Original: D − 12 ˙
Revised: D − 1 2 ˙

Original: A ˙ D − 12 is the normalized adjacency matrix, α ( l ) and β ( l ) are two hyperparameters.
Revised: A ˙ D − 1 2 is the normalized adjacency matrix, α ( l ) and β ( l ) are two hyperparameters.

Original: We utilize GCNII as the backbone that results in the VEM-GCNII architecture.
Revised: We utilize GCNII as the backbone that results in the VEM-GCNII architecture.

Original: We use the ofﬁcial source code from https://github.com/chennnM/GCNII and employ the hyperparameter settings reported in (Chen et al., 2020b) that achieve the best performance on the three citation networks (i.e., Cora, Citeseer, and Pubmed) under the label-scarce setting.
Revised: We use the ofﬁcial source code from https://github.com/chennnM/GCNII and employ the hyperparameter settings reported in (Chen et al., 2020b) that achieve the best performance on the three citation networks (i.e., Cora, Citeseer, and Pubmed) under the label-scarce setting.

Original: For the other four datasets, we roughly tuned the hyperparameters but found that GCNII does not outperform the vanilla GCN.
Revised: For the other four datasets, we roughly tuned the hyperparameters but found that GCNII does not outperform the vanilla GCN.

Original: Thus, we only evaluate GCNII and VEM-GCNII on Cora, Citeseer, and Pubmed with 10 labeled nodes per class as the training set.
Revised: Thus, we only evaluate GCNII and VEM-GCNII on Cora, Citeseer, and Pubmed with 10 labeled nodes per class as the training set.

Original: Experimental results are shown in Table 7.[Table]
Revised: Experimental results are shown in Table 7.