Original: VEM
Revised: VEM

Original: GCN: T OPOLOGY O PTIMIZATION WITH V ARIATIONAL EM FOR G RAPH C ONVOLUTIONAL N ETWORKS
Revised: GCN: T OPOLOGY O PTIMIZATION WITH V ARIATIONAL EM FOR G RAPH C ONVOLUTIONAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Over-smoothing has emerged as a severe problem for node classiﬁcation with graph convolutional networks (GCNs).
Revised: Over-smoothing has emerged as a severe problem for node classiﬁcation with graph convolutional networks (GCNs).

Original: In the view of message passing, the oversmoothing issue is caused by the observed noisy graph topology that would propagate information along inter-class edges, and consequently, over-mix the features of nodes in different classes.
Revised: In the view of message passing, the oversmoothing issue is caused by the observed noisy graph topology that would propagate information along inter-class edges, and consequently, over-mix the features of nodes in different classes.

Original: In this paper, we propose a novel architecture, namely VEM-GCN, to address this problem by employing the variational EM algorithm to jointly optimize the graph topology and learn desirable node representations for classiﬁcation.
Revised: In this paper, we propose a novel architecture, namely VEM-GCN, to address this problem by employing the variational EM algorithm to jointly optimize the graph topology and learn desirable node representations for classiﬁcation.

Original: Speciﬁcally, variational EM approaches a latent adjacency matrix parameterized by the assortative-constrained stochastic block model (SBM) to enhance intra-class connection and suppress inter-class interaction of the observed noisy graph.
Revised: Speciﬁcally, variational EM approaches a latent adjacency matrix parameterized by the assortative-constrained stochastic block model (SBM) to enhance intra-class connection and suppress inter-class interaction of the observed noisy graph.

Original: In the variational E-step, graph topology is optimized by approximating the posterior probability distribution of the latent adjacency matrix with a neural network learned from node embeddings.
Revised: In the variational E-step, graph topology is optimized by approximating the posterior probability distribution of the latent adjacency matrix with a neural network learned from node embeddings.

Original: In the M-step, node representations are learned using the graph convolutional network based on the reﬁned graph topology for the downstream task of classiﬁcation.
Revised: In the M-step, node representations are learned using the graph convolutional network based on the reﬁned graph topology for the downstream task of classiﬁcation.

Original: VEM-GCN is demonstrated to outperform existing strategies for tackling over-smoothing and optimizing graph topology in node classiﬁcation on seven benchmark datasets.
Revised: VEM-GCN is demonstrated to outperform existing strategies for tackling over-smoothing and optimizing graph topology in node classiﬁcation on seven benchmark datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Complex graph-structured data are ubiquitous in the real world, ranging from social networks to chemical molecules.
Revised: Complex graph-structured data are ubiquitous in the real world, ranging from social networks to chemical molecules.

Original: Inspired by the remarkable performance of convolutional neural networks (CNNs) in processing data with regular grid structures (e.g., images), a myriad of studies on GCNs have emerged to execute “convolution” in the graph domain (Niepert et al., 2016; Kipf & Welling, 2017; Gilmer et al., 2017; Hamilton et al., 2017; Monti et al., 2017; Gao et al., 2018).
Revised: Inspired by the remarkable performance of convolutional neural networks (CNNs) in processing data with regular grid structures (e.g., images), a myriad of studies on GCNs have emerged to execute “convolution” in the graph domain (Niepert et al., 2016; Kipf & Welling, 2017; Gilmer et al., 2017; Hamilton et al., 2017; Monti et al., 2017; Gao et al., 2018).

Original: Many of these approaches follow a neighborhood aggregation mechanism (a.k.a., message passing scheme) that updates the representation of each node by iteratively aggregating the transformed messages sent from its neighboring nodes.
Revised: Many of these approaches follow a neighborhood aggregation mechanism (a.k.a., message passing scheme) that updates the representation of each node by iteratively aggregating the transformed messages sent from its neighboring nodes.

Original: Commencing with the pioneering works (Kipf & Welling, 2017;
Revised: Commencing with the pioneering works (Kipf & Welling, 2017;

Original: Gilmer et al., 2017), numerous strategies have been developed to improve the vanilla message passing scheme such as introducing self-attention mechanism (Veliˇckovi´c et al., 2018; Zhang et al., 2020), incorporating local structural information (Zhang et al., 2020;
Revised: Gilmer et al., 2017), numerous strategies have been developed to improve the vanilla message passing scheme such as introducing self-attention mechanism (Veliˇckovi´c et al., 2018; Zhang et al., 2020), incorporating local structural information (Zhang et al., 2020;

Original: Jin et al., 2019;
Revised: Jin et al., 2019;

Original: Ye et al., 2020), and leveraging the link attributes (Gong & Cheng, 2019; Li et al., 2019; Jiang et al., 2019).
Revised: Ye et al., 2020), and leveraging the link attributes (Gong & Cheng, 2019; Li et al., 2019; Jiang et al., 2019).

Original: Despite signiﬁcant success in many fundamental tasks of graph-based machine learning, message passing-based GCNs almost all process the observed graph structure as ground truth and might suffer from the over-smoothing problem (Li et al., 2018), which would seriously affect the node classiﬁcation performance.
Revised: Despite signiﬁcant success in many fundamental tasks of graph-based machine learning, message passing-based GCNs almost all process the observed graph structure as ground truth and might suffer from the over-smoothing problem (Li et al., 2018), which would seriously affect the node classiﬁcation performance.

Original: Given the observed noisy graph topology (i.e., excessive inter-class edges are linked while many intra-class edges are missing), when multiple message passing layers are stacked to enlarge the receptive ﬁeld (the maximum hop of neighborhoods), features of neighboring nodes in different classes would be dominant in message passing.
Revised: Given the observed noisy graph topology (i.e., excessive inter-class edges are linked while many intra-class edges are missing), when multiple message passing layers are stacked to enlarge the receptive ﬁeld (the maximum hop of neighborhoods), features of neighboring nodes in different classes would be dominant in message passing.

Original: Thus, node representations would be corrupted by the harmful noise and affect the discrimination of graph nodes.
Revised: Thus, node representations would be corrupted by the harmful noise and affect the discrimination of graph nodes.

Original: The over-smoothing phenomenon in GCNs has already been studied from different aspects.
Revised: The over-smoothing phenomenon in GCNs has already been studied from different aspects.

Original: Li et al.
Revised: Li et al.

Original: ﬁrst interpreted over-smoothing from the perspective of Laplacian smoothing, while Xu et al.
Revised: ﬁrst interpreted over-smoothing from the perspective of Laplacian smoothing, while Xu et al.

Original: and
Revised: and

Original: Klicpera et al.
Revised: Klicpera et al.

Original: (2019a) associated it with the limit distribution of random walk.
Revised: (2019a) associated it with the limit distribution of random walk.

Original: Furthermore, Chen et al.
Revised: Furthermore, Chen et al.

Original: (2020a) developed quantitative metrics to measure the over-smoothness from the topological view.
Revised: (2020a) developed quantitative metrics to measure the over-smoothness from the topological view.

Original: They argued that the key factor leading to over-smoothing is the noise passing between nodes of different categories and the classiﬁcation performance of GCNs is positively correlated with the proportion of intra-class node pairs in all edges.
Revised: They argued that the key factor leading to over-smoothing is the noise passing between nodes of different categories and the classiﬁcation performance of GCNs is positively correlated with the proportion of intra-class node pairs in all edges.

Original: In this paper, we propose VEM-GCN, a novel architecture to address the over-smoothing problem with topology optimization for uncertain graphs.
Revised: In this paper, we propose VEM-GCN, a novel architecture to address the over-smoothing problem with topology optimization for uncertain graphs.

Original: Considering that a “clearer” graph with more intraclass edges and fewer inter-class edges would improve the node classiﬁcation performance of GCNs (Yang et al., 2019; Chen et al., 2020a), VEM-GCN approaches a latent adjacency matrix parameterized by an assortative-constrained stochastic block model (SBM) where nodes share the same label are linked and inter-class edges should be cut off.
Revised: Considering that a “clearer” graph with more intra-class edges and fewer inter-class edges would improve the node classiﬁcation performance of GCNs (Yang et al., 2019; Chen et al., 2020a), VEM-GCN approaches a latent adjacency matrix parameterized by the assortative-constrained stochastic block model (SBM) where nodes share the same label are linked and inter-class edges should be cut off.

Original: To jointly reﬁne the latent graph structure and learn desirable node representations for classiﬁcation, variational EM algorithm  is adopted to optimize the evidence lower bound (ELBO) of the likelihood function.
Revised: To jointly reﬁne the latent graph structure and learn desirable node representations for classiﬁcation, variational EM algorithm (Neal & Hinton, 1998) is adopted to optimize the evidence lower bound (ELBO) of the likelihood function.

Original: In the inference procedure (E-step), graph topology is optimized by approximating the posterior probability distribution of the latent adjacency matrix with a neural network learned from node embeddings.
Revised: In the inference procedure (E-step), graph topology is optimized by approximating the posterior probability distribution of the latent adjacency matrix with a neural network learned from node embeddings.

Original: In the learning procedure (M-step), a conventional GCN is trained to maximize the log-likelihood of the observed node labels based on the learned latent graph structure.
Revised: In the learning procedure (M-step), a conventional GCN is trained to maximize the log-likelihood of the observed node labels based on the learned latent graph structure.

Original: The E-step and M-step optimize the graph topology and improve the classiﬁcation of unlabeled nodes in an alternating fashion.
Revised: The E-step and M-step optimize the graph topology and improve the classiﬁcation of unlabeled nodes in an alternating fashion.

Original: The proposed VEM-GCN architecture is ﬂexible and general.
Revised: The proposed VEM-GCN architecture is ﬂexible and general.

Original: In the E-step, the neural network can support arbitrary desirable node embeddings generated by algorithms such as node2vec (Grover &
Revised: In the E-step, the neural network can support arbitrary desirable node embeddings generated by algorithms such as node2vec (Grover &

Original: Leskovec, 2016), struc2vec (Ribeiro et al., 2017), and GCNs, or the raw node attributes.
Revised: Leskovec, 2016), struc2vec (Ribeiro et al., 2017), and GCNs, or the raw node attributes.

Original: The GCN in the M-step can also be substituted with arbitrary graph models.
Revised: The GCN in the M-step can also be substituted with arbitrary graph models.

Original: Furthermore, recent strategies for relieving the over-smoothing issue, i.e., AdaEdge (Chen et al., 2020a) and DropEdge (Rong et al., 2020), are shown to be the speciﬁc cases of VEM-GCN under certain conditions.
Revised: Furthermore, recent strategies for relieving the over-smoothing issue, i.e., AdaEdge (Chen et al., 2020a) and DropEdge (Rong et al., 2020), are shown to be the speciﬁc cases of VEM-GCN under certain conditions.

Original: For empirical evaluation, we conduct extensive experiments on seven benchmarks for node classiﬁcation, including four citation networks, two Amazon co-purchase graphs, and one Microsoft Academic graph.
Revised: For empirical evaluation, we conduct extensive experiments on seven benchmarks for node classiﬁcation, including four citation networks, two Amazon co-purchase graphs, and one Microsoft Academic graph.

Original: Experimental results demonstrate the effectiveness of the proposed VEM-GCN architecture in optimizing graph topology and mitigating the over-smoothing problem for GCNs.
Revised: Experimental results demonstrate the effectiveness of the proposed VEM-GCN architecture in optimizing graph topology and mitigating the over-smoothing problem for GCNs.

Original:  B ACKGROUND AND R ELATED W ORKS
Revised:  B ACKGROUND AND R ELATED W ORKS

Original: Problem Setting.
Revised: Problem Setting.

Original: This paper focuses on the task of graph-based transductive node classiﬁcation.
Revised: This paper focuses on the task of graph-based transductive node classiﬁcation.

Original: A simple attributed graph is deﬁned as a tuple G obs =
Revised: A simple attributed graph is deﬁned as a tuple G obs =

Original: ( V , A obs , X ) , where V = { v i } Ni =1 is the node set, A obs = (cid:2) a obs ij (cid:3) ∈ { 0 , 1 } N × N is the observed adjacency matrix, and X ∈ R N × f represents the collection of attributes with each row corresponding to the features of an individual node.
Revised: ( V , A obs , X ) , where V = { v i } Ni =1 is the node set, A obs = (cid:2) a obs ij (cid:3) ∈ { 0 , 1 } N × N is the observed adjacency matrix, and X ∈ R N × f represents the collection of attributes with each row corresponding to the features of an individual node.

Original: Given the labels Y l =
Revised: Given the labels Y l =

Original: [ y ic ] ∈ { 0 , 1 } |V l |× C for a subset of graph nodes V l ⊂ V assigned to C classes, the task is to infer the classes Y u
Revised: [ y ic ] ∈ { 0 , 1 } |V l |× C for a subset of graph nodes V l ⊂ V assigned to C classes, the task is to infer the classes Y u

Original: =
Revised: =

Original: [ y jc ] ∈ { 0 , 1 } |V u |× C for the unlabeled nodes V u = V\V l based on G obs .
Revised: [ y jc ] ∈ { 0 , 1 } |V u |× C for the unlabeled nodes V u = V\V l based on G obs .

Original: Graph Convolutional Networks (GCNs).
Revised: Graph Convolutional Networks (GCNs).

Original: The core of most GCNs is message passing scheme, where each node updates its representation by iteratively aggregating features from its neighborhoods.
Revised: The core of most GCNs is message passing scheme, where each node updates its representation by iteratively aggregating features from its neighborhoods.

Original: Denote with W ( l ) the learnable weights in the l -th layer, N ( i ) the set of neighboring node indices for node v i , and σ ( · ) the nonlinear activation function.
Revised: Denote with W ( l ) the learnable weights in the l -th layer, N ( i ) the set of neighboring node indices for node v i , and σ ( · ) the nonlinear activation function.

Original: A  message passing layer follows
Revised: A basic message passing layer takes the following form:

Original: [Equation]
Revised: [Equation]

Original: Here, h ( l ) j is the input features of node v j in the l -th layer, W  l  h ( [Equation] ) j is the corresponding transformed message, and α  ij is the aggregation weight for the message passing from node v j to node v i .
Revised: Here, h ( l ) j is the input features of node v j in the l -th layer, W ( l ) h ( l ) j is the corresponding transformed message, and α ( l ) ij is the aggregation weight for the message passing from node v j to node v i .

Original: Existing GCNs mainly differ in the mechanism for computing α  ij (Kipf & Welling, 2017; Veliˇckovi´c et al., 2018;
Revised: Existing GCNs mainly differ in the mechanism for computing α ( l ) ij (Kipf & Welling, 2017; Veliˇckovi´c et al., 2018;

Original: Ye et al., 2020; Hamilton et al., 2017; Zhang et al., 2020).
Revised: Ye et al., 2020; Hamilton et al., 2017; Zhang et al., 2020).

Original: Stochastic Block Model (SBM).
Revised: Stochastic Block Model (SBM).

Original: SBM (Holland et al., 1983) is a generative model for producing graphs with community structures.
Revised: SBM (Holland et al., 1983) is a generative model for producing graphs with community structures.

Original: It parameterizes the edge probability between each node pair by
Revised: It parameterizes the edge probability between each node pair by

Original: [Equation]
Revised: [Equation]

Original: where ¯ a ij is an indicator variable for the edge linking nodes v i and v j , y i and y j denote their corresponding communities (classes), p 0 and p 1 are termed community link strength and crosscommunity link probability, respectively.
Revised: where ¯ a ij is an indicator variable for the edge linking nodes v i and v j , y i and y j denote their corresponding communities (classes), p 0 and p 1 are termed community link strength and cross- community link probability, respectively.

Original: The case where p 0 > p 1 is called an assortative model, while the case p 0 < p 1 is called disassortative.
Revised: The case where p 0 > p 1 is called an assortative model, while the case p 0 < p 1 is called disassortative.

Original: In this paper, we leverage an assortative-constrained
Revised: In this paper, we leverage an assortative-constrained

Original: SBM (Gribel et al., 2020) with p 0 = 1 and p 1 = 0 to model the latent graph for a clear topology.
Revised: SBM (Gribel et al., 2020) with p 0 = 1 and p 1 = 0 to model the latent graph for a clear topology.

Original: Over-smoothing.
Revised: Over-smoothing.

Original: Real-world graphs often possess high sparsity and are corrupted by certain noise that leads to inter-class misconnection and missing intra-class edges.
Revised: Real-world graphs often possess high sparsity and are corrupted by certain noise that leads to inter-class misconnection and missing intra-class edges.

Original: Over-smoothing is mainly caused by the indistinguishable features of nodes in different classes produced by the message passing along inter-class edges.
Revised: Over-smoothing is mainly caused by the indistinguishable features of nodes in different classes produced by the message passing along inter-class edges.

Original: Various strategies have been developed to alleviate this problem.
Revised: Various strategies have been developed to alleviate this problem.

Original: JK-Net (Xu et al., 2018) utilizes skip connection for adaptive feature aggregation and DNA (Fey, 2019) further makes improvements based on the attention mechanism.
Revised: JK-Net (Xu et al., 2018) utilizes skip connection for adaptive feature aggregation and DNA (Fey, 2019) further makes improvements based on the attention mechanism.

Original: PPNP and APPNP (Klicpera et al., 2019a) modify the message passing scheme by personalized PageRank (PPR) to avoid reaching the limit distribution of random walk.
Revised: PPNP and APPNP (Klicpera et al., 2019a) modify the message passing scheme by personalized PageRank (PPR) to avoid reaching the limit distribution of random walk.

Original: CGNN (Xhonneux et al., 2020) addresses over-smoothing in a similar manner as PPR.
Revised: CGNN (Xhonneux et al., 2020) addresses over-smoothing in a similar manner as PPR.

Original: Zhao & Akoglu (2020) introduced a graph layer normalization scheme termed PairNorm to maintain the total pairwise distance between nodes unchanged across layers.
Revised: Zhao & Akoglu (2020) introduced a graph layer normalization scheme termed PairNorm to maintain the total pairwise distance between nodes unchanged across layers.

Original: GCNII (Chen et al., 2020b) extends GCN with Initial residual and Identity mapping.
Revised: GCNII (Chen et al., 2020b) extends GCN with Initial residual and Identity mapping.

Original: However, these methods cannot well address the over-smoothing issue, as they all view the observed graph as ground truth and the features of nodes in different classes would still be over-mixed along the inter-class edges.
Revised: However, these methods cannot fundamentally address the over-smoothing issue, as they all view the observed graph as ground truth and the features of nodes in different classes would still be over-mixed along the inter-class edges.

Original: AdaEdge (Chen et al., 2020a) constantly reﬁnes the graph topology by adjusting the edges in a self-training-like fashion.
Revised: AdaEdge (Chen et al., 2020a) constantly reﬁnes the graph topology by adjusting the edges in a self-training-like fashion.

Original: However, AdaEdge only adjusts the edges linking nodes classiﬁed with high conﬁdence, which leads to limited improvement or degradation in classiﬁcation performance due to the incorrect operations for misclassiﬁed nodes.
Revised: However, AdaEdge only adjusts the edges linking nodes classiﬁed with high conﬁdence, which leads to limited improvement or degradation in classiﬁcation performance due to the incorrect operations for misclassiﬁed nodes.

Original: DropEdge (Rong et al., 2020) randomly removes a certain fraction of edges to reduce message passing.
Revised: DropEdge (Rong et al., 2020) randomly removes a certain fraction of edges to reduce message passing.

Original: Despite enhanced robustness, DropEdge does not essentially optimize the graph topology.
Revised: Despite enhanced robustness, DropEdge does not essentially optimize the graph topology.

Original: BBGDC (Hasanzadeh et al., 2020) generalizes Dropout (Srivastava et al., 2014) and DropEdge by adaptive connection sampling.
Revised: BBGDC (Hasanzadeh et al., 2020) generalizes Dropout (Srivastava et al., 2014) and DropEdge by adaptive connection sampling.

Original: Uncertain Graphs and Topology Optimization.
Revised: Uncertain Graphs and Topology Optimization.

Original: Learning with uncertain graphs is another related research area, where the observed graph structure is supposed to be derived from noisy data rather than ground truth.
Revised: Learning with uncertain graphs is another related research area, where the observed graph structure is supposed to be derived from noisy data rather than ground truth.

Original: Bayesian approaches are typical methods that introduce uncertainty to network analysis.
Revised: Bayesian approaches are typical methods that introduce uncertainty to network analysis.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2019) developed BGCN that considers the observed graph as a sample from a parametric family of random graphs and makes maximum a posteriori (MAP) estimate of the graph parameters.
Revised: (2019) developed BGCN that considers the observed graph as a sample from a parametric family of random graphs and makes maximum a posteriori (MAP) estimate of the graph parameters.

Original: Tiao et al.
Revised: Tiao et al.

Original: (2019) also viewed graph edges as Bernoulli random variables and used variational inference to optimize the posterior distribution of the adjacency matrix by approximating the pre-deﬁned graph priors.
Revised: (2019) also viewed graph edges as Bernoulli random variables and used variational inference to optimize the posterior distribution of the adjacency matrix by approximating the pre-deﬁned graph priors.

Original: Some other Bayesian methods have also been developed to combine GCNs with probabilistic models (Ng et al., 2018; Ma et al., 2019).
Revised: Some other Bayesian methods have also been developed to combine GCNs with probabilistic models (Ng et al., 2018; Ma et al., 2019).

Original: However, without explicit optimization for the graph structure, they only improve the robustness under certain conditions such as incomplete edges, active learning, and adversarial attacks.
Revised: However, without explicit optimization for the graph structure, they only improve the robustness under certain conditions such as incomplete edges, active learning, and adversarial attacks.

Original: For explicit topology optimization, Franceschi et al.
Revised: For explicit topology optimization, Franceschi et al.

Original: (2019) presented LDS to parameterize edges as independent Bernoulli random variables and learn discrete structures for GCNs by solving a bilevel programming.
Revised: (2019) presented LDS to parameterize edges as independent Bernoulli random variables and learn discrete structures for GCNs by solving a bilevel programming.

Original: However, LDS requires an extra validation set for training and suffers from limited scalability.
Revised: However, LDS requires an extra validation set for training and suffers from limited scalability.

Original: TO-GCN (Yang et al., 2019) only adds the intra-class edges derived from the labeled nodes, which causes topology imbalance between V u and V l .
Revised: TO-GCN (Yang et al., 2019) only adds the intra-class edges derived from the labeled nodes, which causes topology imbalance between V u and V l .

Original: GDC (Klicpera et al., 2019b) reﬁnes the adjacency matrix with graph diffusion to consider the links between high-order neighborhoods.
Revised: GDC (Klicpera et al., 2019b) reﬁnes the adjacency matrix with graph diffusion to consider the links between high-order neighborhoods.

Original: However, the added edges might still be noisy to hamper the classiﬁcation. 
Revised: However, the added edges might still be noisy to hamper the classiﬁcation. GRCN

Revised: (Yu et al., 2020) modiﬁes the original adjacency matrix by adding a residual matrix with each element measuring the similarity between two corresponding node embeddings, and IDGL (Chen et al., 2020c) iteratively learns the graph structure in a similar manner.

Revised: Pro-GNN (Jin et al., 2020) introduces low rank and sparsity constraints to recover a clean graph in defending adversarial attacks.

Revised: NeuralSparse (Zheng et al., 2020) uses the Gumbel Softmax trick (Jang et al., 2017) to sample k neighbors from the original neighborhoods for each node but does not consider recovering missing intra-class edges.

Original: Different from the aforementioned methods, VEM-GCN relieves over-smoothing by introducing a learned latent graph based on the assortative-constrained SBM to explicitly enhance intra-class connection and suppress inter-class interaction with  variational EM.
Revised: Different from the aforementioned methods, VEM-GCN aims at relieving the over-smoothing issue. We introduce a learned latent graph based on the assortative-constrained SBM to explicitly enhance intra-class connection and suppress inter-class interaction with the variational EM algorithm.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In this section, we develop the VEM-GCN architecture for transductive node classiﬁcation.
Revised: In this section, we develop the VEM-GCN architecture for transductive node classiﬁcation.

Original: VEMGCN leverages the variational EM algorithm to achieve topology optimization, and consequently, address the over-smoothing problem by reducing noisy interactions between nodes in different classes.
Revised: VEMGCN leverages the variational EM algorithm to achieve topology optimization, and consequently, address the over-smoothing issue by reducing noisy interactions between nodes in different classes.

Original: Speciﬁcally, E-step approximates the posterior probability distribution of the latent adjacency matrix to optimize the graph structure, and M-step maximizes the evidence lower bound of the log-likelihood function based on the reﬁned graph.
Revised: Speciﬁcally, E-step approximates the posterior probability distribution of the latent adjacency ma- trix to optimize the graph structure, and M-step maximizes the evidence lower bound of the loglikelihood function based on the reﬁned graph.

Original: We ﬁrst introduce our motivation and provide an overview of the proposed VEM-GCN architecture.
Revised: We ﬁrst introduce our motivation and provide an overview of the proposed VEM-GCN architecture.

Original: Subsequently, we elaborate the mechanisms of the variational E-step and M-step, respectively.
Revised: Subsequently, we elaborate the mechanisms of the variational E-step and M-step, respectively.

Original: 3. M OTIVATION AND O VERVIEW
Revised: 3. M OTIVATION AND O VERVIEW

Original: Motivation.
Revised: Motivation.

Original: As mentioned above, a graph with its nodes densely connected within their own communities (classes) has lower risk of over-smoothing.
Revised: As mentioned above, a graph with its nodes densely connected within their own communities (classes) has lower risk of over-smoothing.

Original: Under this consideration, the optimal adjacency matrix for GCN is ˜ A
Revised: Under this consideration, the optimal adjacency matrix for GCN is ˜ A

Original: =
Revised: =

Original: YY (cid:62) (Yang et al., 2019; Chen et al., 2020a), where Y ∈ R N × C is the matrix of one-hot-encoded ground-truth labels.
Revised: YY (cid:62) (Yang et al., 2019; Chen et al., 2020a), where Y ∈ R N × C is the matrix of one-hot-encoded ground-truth labels.

Original: However, since we have to infer Y u for the unlabeled nodes V u , their true labels are not available for calculating ˜ A .
Revised: However, since we have to infer Y u for the unlabeled nodes V u , their true labels are not available for calculating ˜ A .

Original: Thus, we introduce a latent graph A latent learned from G obs with anotherauxiliary neural network that helps generate a topology clearer than A obs for GCNs.
Revised: Thus, we introduce a latent graph A latent learned from G obs through another neural network to help generate a topology clearer than A obs for GCNs.

Original: It is obvious that ˜ A is equivalent to a SBM with p 0 = 1 and p 1 = 0 , and therefore we base the posterior probability distribution of the latent graph on this assumption.
Revised: It is obvious that ˜ A is equivalent to a SBM with p 0 = 1 and p 1 = 0 , and therefore we base the posterior probability distribution of the latent graph on this assumption.

Original: Overview.
Revised: Overview.

Original: The basic principle behind our proposed VEM-GCN architecture is maximum likelihood estimation (MLE) in a latent variable model, i.e., to maximize the log-likelihood function of the observed node labels E q φ ( A latent |G obs ) [log p θ ( Y l |G obs )] based on the approximate posterior distribution q φ ( A latent |G obs ) of the latent graph A latent .
Revised: The basic principle behind our proposed VEM-GCN architecture is maximum likelihood estimation (MLE) in a latent variable model, i.e., to maximize the log-likelihood function of the observed node labels E q φ ( A latent |G obs ) [log p θ ( Y l |G obs )] based on the approximate posterior distribution q φ ( A latent |G obs ) of the latent graph A latent .

Original: According to variational inference, the evidence lower bound (ELBO) is optimized instead:
Revised: According to variational inference, the evidence lower bound (ELBO) is optimized instead:

Original: [Equation]
Revised: [Equation]

Original: where the equality holds when q φ ( A latent |G obs )
Revised: where the equality holds when q φ ( A latent |G obs )

Original: = p θ ( A latent | Y l , G obs ) .
Revised: = p θ ( A latent | Y l , G obs ) .

Original: Note that q φ can be arbitrary desirable distributions on A latent and we use a neural network to parameterize it in this work.
Revised: Note that q φ can be arbitrary desirable distributions on A latent and we use a neural network to parameterize it in this work.

Original: To jointly optimize the latent graph topology A latent and the ELBO L ELBO ( θ, φ ; Y l , G obs ) , we adopt the variational EM algorithm to solve it (refer to Appendix A for the full algorithm).
Revised: To jointly optimize the latent graph topology A latent and the ELBO L ELBO ( θ, φ ; Y l , G obs ) , we adopt the variational EM algorithm to solve it (refer to Appendix A for the full algorithm).

Original: 3. E- STEP
Revised: 3. E- STEP

Original: In the inference procedure (E-step), θ is ﬁxed and the goal is to optimze q φ ( A latent |G obs ) to approximate the true posterior distribution p θ ( A latent | Y l , G obs ) .
Revised: In the inference procedure (E-step), θ is ﬁxed and the goal is to optimze q φ ( A latent |G obs ) to approximate the true posterior distribution p θ ( A latent | Y l , G obs ) .

Original: Under the condition of SBM, we assume each edge of the latent graph to be independent.
Revised: Under the condition of SBM, we assume each edge of the latent graph to be independent.

Original: Thus, q φ ( A latent |G obs ) can be factorized by:
Revised: Thus, q φ ( A latent |G obs ) can be factorized by:

Original: [Equation]
Revised: [Equation]

Original: Unlike LDS (Franceschi et al., 2019) using O ( N 2 ) Bernoulli random variables to characterize the optimized graph with N nodes, we parameterize q φ ( a latent ij |G obs ) through a neural network shared by all the possible node pairs (i.e., amortized variational inference (Gershman & Goodman, 2014)), as shown in Eq.
Revised: Unlike LDS (Franceschi et al., 2019) using O ( N 2 ) Bernoulli random variables to characterize the optimized graph with N nodes, we parameterize q φ ( a latent ij |G obs ) through a neural network shared by all the possible node pairs (i.e., amortized variational inference (Gershman & Goodman, 2014)), as shown in Eq.

Original: Hence, our method shows scalability for large-scale graphs and is easier to train.
Revised: Hence, our method shows scalability for large-scale graphs and is easier to train.

Original: [Equation]
Revised: [Equation]

Original:  where e i is the node embedding of node v i , which can be derived from any desirable network embedding methods (e.g., node2vec (Grover & Leskovec, 2016), struc2vec (Ribeiro et al., 2017), and GCNs) or the raw node attributes x i (the i -th row of X ), and z i is the transformed features of node v i .
Revised: [Equation] ) where e i is the node embedding of node v i , which can be derived from any desirable network embedding methods (e.g., node2vec (Grover & Leskovec, 2016), struc2vec (Ribeiro et al., 2017), and GCNs) or the raw node attributes x i (the i -th row of X ), and z i is the transformed features of node v i .

Original: NN ( · ) denotes a neural network and we use a Multi-Layer Perceptron (MLP) in this work.
Revised: NN ( · ) denotes a neural network and we use a Multi-Layer Perceptron (MLP) in this work.

Original: The probability for linking a node pair is deﬁned as the inner-product of their transformed features activated by a sigmoid function.
Revised: The probability for linking a node pair is deﬁned as the inner-product of their transformed features activated by a sigmoid function.

Original: To approximate the posterior probability distribution of A latent , we rewrite p θ ( A latent | Y l , G obs ) as:
Revised: To approximate the posterior probability distribution of A latent , we rewrite p θ ( A latent | Y l , G obs ) as:

Original: [Equation]
Revised: [Equation]

Original: , G obs )] . [Equation] Here, p θ ( A latent | Y l , Y u , G obs ) is parameterized by the aforementioned assortative-constrained SBM (i.e., p θ ( a latent ij = [Equation]
Revised: Here, p θ ( A latent | Y l , Y u , G obs ) is parameterized by the aforementioned assortative-constrained SBM (i.e., p θ ( a latent ij = 1 | y i , y j ) =

Original: y i, y [Equation] ) = y i y (cid:62) j for the one-hot-encoded node label y ), p θ ( Y u | Y l , G obs ) is the predicted categorical distributions for the unlabeled nodes derived in the previous M-step.
Revised: y i y (cid:62) j for the one-hot-encoded node label y ), p θ ( Y u | Y l , G obs ) is the predicted categorical distributions for the unlabeled nodes derived in the previous M-step.

Original: Consequently, we can sample ˆ Y u ∼ p θ ( Y u | Y l , G obs ) to estimate the expectation in Eq.
Revised: Consequently, we can sample ˆ Y u ∼ p θ ( Y u | Y l , G obs ) to estimate the expectation in Eq.

Original: 6 and leverage stochastic gradient descent (SGD) to minimize the reverse KL-divergence between the approximate posterior distribution q φ ( A latent |G obs ) and the target p θ
Revised: 6 and leverage stochastic gradient descent (SGD) to minimize the reverse KL-divergence between the approximate posterior distribution q φ ( A latent |G obs ) and the target p θ

Original: ( A latent | Y l , G obs ) .
Revised: ( A latent | Y l , G obs ) .

Original: Under appropriate assumptions, q φ will converge to p θ ( A latent | Y l , G obs ) as the iteration step of SGD t → ∞ (Bottou, 2010).
Revised: Under appropriate assumptions, q φ will converge to p θ ( A latent | Y l , G obs ) as the iteration step of SGD t → ∞ (Bottou, 2010).

Original: Thus, we can obtain the following objective function in the variational E-step for optimizing φ :
Revised: Thus, we can obtain the following objective function in the variational E-step for optimizing φ :

Original: [Equation]
Revised: [Equation]

Original: where y is the ground truth label for node in labeled set V l , otherwise sampled from p
Revised: where y is the ground truth label for node in labeled set V l , otherwise sampled from p

Original: θ ( Y u | Y l , G obs ) for the nodes without given labels in each training step, and λ ( a latent ij ) is the weighting hyperparameter to alleviate class imbalance between the inter-class edges and the intra-class edges.
Revised: θ ( Y u | Y l , G obs ) for the nodes without given labels in each training step, and λ ( a latent ij ) is the weighting hyperparameter to alleviate class imbalance between the inter-class edges and the intra-class edges.

Original: 3. M- STEP
Revised: 3. M- STEP

Original: In the learning procedure (M-step), φ is ﬁxed and θ is updated to maximize the ELBO in Eq.
Revised: In the learning procedure (M-step), φ is ﬁxed and θ is updated to maximize the ELBO in Eq.

Original: By factorizing p θ ( Y l , A latent |G obs ) = p
Revised: By factorizing p θ ( Y l , A latent |G obs ) = p

Original: [Equation]
Revised: [Equation]

Revised: ( Y l | A

Original: 1 ( Y l | A latent , G obs ) p θ 2 ( A latent |G obs ) with θ = { θ 1 , θ 2 } , we have:
Revised: [Equation] , G obs ) p [Equation] 2 ( A latent |G obs ) with θ = { θ 1 , θ 2 } , we have:

Original: [Equation]
Revised: [Equation]

Original: Here, p θ 1 ( Y l | A latent , G obs ) in the ﬁrst term can be parameterized by arbitrary GCN models described by Eq. 1 that infer the node labels from A latent and X .
Revised: Here, p θ 1 ( Y l | A latent , G obs ) in the ﬁrst term can be parameterized by arbitrary GCN models described by Eq. 1 that infer the node labels from A latent and X .

Original: We use the vanilla GCN (Kipf & Welling, 2017) in this work (see Eq. 13 in Appendix A).
Revised: We use the vanilla GCN (Kipf & Welling, 2017) in this work (see Eq. 13 in Appendix A).

Original: The second term is the KL-divergence between q φ ( A latent |G obs ) and the prior p θ 2 ( A latent |G obs ) , which can be optimized by setting θ 2 = φ to force
Revised: The second term is the KL-divergence between q φ ( A latent |G obs ) and the prior p θ 2 ( A latent |G obs ) , which can be optimized by setting θ 2 = φ to force

Original: KL[ q φ ( A latent |G obs ) (cid:107) p θ 

Revised: KL[ q φ ( A latent |G obs ) (cid:107) p θ

Original: A latent |G obs )]
Revised: A latent |G obs )]

Original: = 0 .
Revised: = 0 .

Original: Actually, p θ 2 ( A latent |G obs ) is of little interest to the ﬁnal  classiﬁcation task and we just need to maximize E qφ ( A latent | X ) [log p θ 1 ( Y l
Revised: Actually, p θ 2 ( A latent |G obs ) is of little interest to the ﬁnal node classiﬁcation task and we just need to maximize E q

Original: | A latent , X )] in the M-step.
Revised: [Equation] ] in the M-step.

Original: Considering the fact that the observed graph structure A obs should not be fully discarded and the approximation q φ ( A latent |G obs ) derived in the previous E-step is sometimes not very accurate, we use q φ ( A latent |G obs ) to reﬁne A obs , and substitute q φ with the following ¯ q φ in practice:
Revised: Considering the fact that the observed graph structure A obs should not be fully discarded and the approximation q φ ( A latent |G obs ) derived in the previous E-step is sometimes not very accurate, we use q φ ( A latent |G obs ) to reﬁne A obs , and substitute q φ with the following ¯ q φ in practice:

Original: [Equation]
Revised: [Equation]

Original: where p ∈ (0 , 1] , ε 1 is close to one (commonly 0 . 999 ), and ε 2 is close to zero (commonly 0 . 01 ).
Revised: where p ∈ (0 , 1] , ε 1 is close to one (commonly 0 . 999 ), and ε 2 is close to zero (commonly 0 . 01 ).

Original: Eq.
Revised: Eq.

Original: 9 implies that, for edges predicted by q φ to be linked with high conﬁdence (the value after sigmoid or the maximum value after softmax), they should be added to the observed graph with probability p .
Revised: 9 implies that, for edges predicted by q φ to be linked with high conﬁdence (the value after sigmoid or the maximum value after softmax), they should be added to the observed graph with probability p .

Original: Edges predicted by q φ to be cut off with high conﬁdence should be removed from the observed graph.
Revised: Edges predicted by q φ to be cut off with high conﬁdence should be removed from the observed graph.

Original: Otherwise, we maintain the original graph structure with probability p .
Revised: Otherwise, we maintain the original graph structure with probability p .

Original: Similar to the E-step, we can sample the latent adjacency matrix ˆ A latent ∼ ¯
Revised: Similar to the E-step, we can sample the latent adjacency matrix ˆ A latent ∼ ¯

Original: q φ ( A latent |G obs ) (note that we pre-train p θ 1 using A obs ) and leverage SGD to minimize the cross-entropy error between the
Revised: q φ ( A latent |G obs ) (note that we pre-train p θ 1 using A obs ) and leverage SGD to minimize the cross-entropy error between the

Original: GCN’s predictions p θ 1 ( Y l | ˆ A latent , G obs )] and the ground-truth labels Y l for optimizing θ :
Revised: GCN’s predictions p θ 1 ( Y l | ˆ A latent , G obs ) and the ground-truth labels Y l for optimizing θ :

Original: [Equation]
Revised: [Equation]

Original: In the test procedure, the ﬁnal predictions for Y u are E ¯ q
Revised: In the test procedure, the ﬁnal predictions for Y u are E ¯ q

Original: [Equation]
Revised: [Equation]

Original: latent
Revised: obs

Original: [Equation]
Revised: [Equation]

Revised: [ p

Original: [Equation]
Revised: [Equation]

Original:  Y u |
Revised: ( Y u |

Original: [Equation] , G obs )] , which can be approximated by Monte-Carlo sampling:[Equation]
Revised: A latent , G obs )] , which can be approximated by Monte-Carlo sampling:

Original: [Equation]
Revised: [Equation]

Original: where the number of samples S and the probability p in ¯ q φ are tuned hyperparameters.
Revised: where the number of samples S and the probability p in ¯ q φ are tuned hyperparameters.

Original: The two neural networks q φ and p θ are trained in an alternating fashion to reinforce each other.
Revised: The two neural networks q φ and p θ are trained in an alternating fashion to reinforce each other.

Original:  in the E-step improves the performance of the GCN in the M-step, and with more unlabeled nodes being correctly classiﬁed, q φ will better approximate the optimal graph ˜ A .
Revised: Topology optimization in the E-step improves the performance of the GCN in the M-step, and with more unlabeled nodes being correctly classiﬁed, q φ will better approximate the optimal graph ˜ A .

Original: 3. D ISCUSSIONS
Revised: 3. D ISCUSSIONS

Original: In this subsection, we discuss the relationship between VEM-GCN and two recent works for tackling over-smoothing, i.e., DropEdge (Rong et al., 2020) and AdaEdge (Chen et al., 2020a).
Revised: In this subsection, we discuss the relationship between VEM-GCN and two recent works for tackling over-smoothing, i.e., DropEdge (Rong et al., 2020) and AdaEdge (Chen et al., 2020a).

Original: We show that these two methods are speciﬁc cases of VEM-GCN under certain conditions.
Revised: We show that these two methods are speciﬁc cases of VEM-GCN under certain conditions.

Original: More detailed comparisons with other related works (e.g., SBM-related GCNs) are discussed in Appendix B.
Revised: More detailed comparisons with other related works (e.g., SBM-related GCNs) are discussed in Appendix B.

Original: VEM-GCN vs. DropEdge.
Revised: VEM-GCN vs. DropEdge.

Original: DropEdge randomly removes a certain fraction of edges in each training step.
Revised: DropEdge randomly removes a certain fraction of edges in each training step.

Original: The authors proved that this strategy can retard the convergence speed of over-smoothing.
Revised: The authors proved that this strategy can retard the convergence speed of over-smoothing.

Original: However, it does not address the over-smoothing issue at the core, since the graph topology is not fundamentally optimized and noisy messages still pass along inter-class edges.
Revised: However, it does not address the over-smoothing issue at the core, since the graph topology is not fundamentally optimized and noisy messages still pass along inter-class edges.

Original: Considering the scenario where a node has few interactions with its community but many cross-community links,
Revised: Considering the scenario where a node has few interactions with its community but many cross-community links,

Original: DropEdge cannot improve the discrimination of this stray node, since it does not recover the missing intra-class edges.
Revised: DropEdge cannot improve the discrimination of this stray node, since it does not recover the missing intra-class edges.

Original: We ﬁnd that VEM-GCN degenerates to DropEdge, if we skip the E-step and just maximize E ¯ q φ ( A latent |G obs ) [log p θ 1 ( Y l | A latent , G obs )] with ¯ q φ ( a latent ij = 1 |G obs ) = p · a obs ij .
Revised: We ﬁnd that VEM-GCN degenerates to DropEdge, if we skip the E-step and just maximize E ¯ q φ ( A latent |G obs ) [log p θ 1 ( Y l | A latent , G obs )] with ¯ q φ ( a latent ij = 1 |G obs ) = p · a obs ij .

Original: VEM-GCN vs. AdaEdge.
Revised: VEM-GCN vs. AdaEdge.

Original: AdaEdge also constantly adjusts the graph topology in the training procedure.
Revised: AdaEdge also constantly adjusts the graph topology in the training procedure.

Original: It adds the edge between two nodes which are predicted by the GCN as the same class with high conﬁdence, and removes edges in a similar manner.
Revised: It adds the edge between two nodes which are predicted by the GCN as the same class with high conﬁdence, and removes edges in a similar manner.

Original: If we skip the E-step and set ¯ q φ as
Revised: If we skip the E-step and set ¯ q φ as

Original: Eq.
Revised: Eq.

Original: 12, VEM-GCN and AdaEdge can be equivalent.
Revised: 12, VEM-GCN and AdaEdge can be equivalent.

Original: [Equation]
Revised: [Equation]

Original:  where y (cid:48) is the prediction made by GCN, conf( · ) denotes the corresponding conﬁdence, τ 1 and τare two thresholds.
Revised: , if y (cid:48) i (cid:54) = y (cid:48) j and conf( y (cid:48) i ) , conf( y (cid:48) j ) > τa obs ij , otherwise , (12) where y (cid:48) is the prediction made by GCN, conf( · ) denotes the corresponding conﬁdence, τ 1 and τare two thresholds.

Original: Eq.
Revised: Eq.

Original: 12 implies that, this self-training-like fashion only adjusts the edges whose interacting nodes have already been classiﬁed with high conﬁdence.
Revised: 12 implies that, this self-training-like fashion only adjusts the edges whose interacting nodes have already been classiﬁed with high conﬁdence.

Original: Therefore, the performance improvement is limited and would even get worse for some misclassiﬁed nodes, as it might wrongly add inter-class edges to the observed graph A obs and remove helpful intra-class connections.
Revised: Therefore, the performance improvement is limited and would even get worse for some misclassiﬁed nodes, as it might wrongly add inter-class edges to the observed graph A obs and remove helpful intra-class connections.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To evaluate our VEM-GCN architecture, we conduct extensive experiments on seven benchmark datasets.
Revised: To evaluate our VEM-GCN architecture, we conduct extensive experiments on seven benchmark datasets.

Original: Under the same setting as DropEdge (Rong et al., 2020) and a label-scarce setting (i.e., low label rate), we compare the performance of VEM-GCN against a variety of state of the arts for tackling over-smoothing, uncertain graphs and topology optimization in GCNs.
Revised: Under the same setting as DropEdge (Rong et al., 2020) and a label-scarce setting (i.e., low label rate), we compare the performance of VEM-GCN against a variety of state of the arts for tackling over-smoothing, uncertain graphs and topology optimization in GCNs.

Original: We further give the visualization results of topology optimization and quantitative analysis to verify the effectiveness of
Revised: We further give the visualization results of topology optimization and quantitative analysis to verify the effectiveness of

Original: VEM-GCN in relieving the over-smoothing issue (complexity analysis is provided in Appendix E.3).
Revised: VEM-GCN in relieving the over-smoothing issue (complexity analysis is provided in Appendix E.3).

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: Datasets and Baselines.
Revised: Datasets and Baselines.

Original: We adopt seven well-known benchmark datasets to validate the proposed method.
Revised: We adopt seven well-known benchmark datasets to validate the proposed method.

Original: Cora (Sen et al., 2008), Cora-ML (McCallum et al., 2000; Bojchevski & G ¨ unnemann, 2018), Citeseer (Sen et al., 2008), and Pubmed (Namata et al., 2012) are four citation network benchmarks, where nodes represent documents and edges are citations between documents.
Revised: Cora (Sen et al., 2008), Cora-ML (McCallum et al., 2000; Bojchevski & G¨unnemann, 2018), Citeseer (Sen et al., 2008), and Pubmed (Namata et al., 2012) are four citation network benchmarks, where nodes represent documents and edges are citations between documents.

Original: Amazon
Revised: Amazon

Original: Photo and Amazon Computers are two segments from the Amazon co-purchase graph (McAuley et al., 2015), in which nodes represent goods and edges indicate that two goods are frequently bought together.
Revised: Photo and Amazon Computers are two segments from the Amazon co-purchase graph (McAuley et al., 2015), in which nodes represent goods and edges indicate that two goods are frequently bought together.

Original: In the Microsoft Academic graph (Shchur et al., 2018), nodes are authors and edges represent their co-authorship.
Revised: In the Microsoft Academic graph (Shchur et al., 2018), nodes are authors and edges represent their co-authorship.

Original: All graphs use bag-of-words encoded representations as node attributes.
Revised: All graphs use bag-of-words encoded representations as node attributes.

Original: An overview of the dataset statistics is summarized in Appendix C.
Revised: An overview of the dataset statistics is summarized in Appendix C.

Original: Since VEM-GCN aims at addressing the over-smoothing problem withgraph topology optimization, we evaluate the node classiﬁcation performance of our method against various strategies foraddressing the over-smoothing issue, tackling  uncertain graphs and topology optimization in GCNs.
Revised: Since VEM-GCN aims at addressing the over-smoothing problem with topology optimization, we evaluate the node classiﬁcation performance of our method against various strategies for tackling over-smoothing, uncertain graphs and topology optimization in GCNs.

Original: For addressing the over-smoothing issue, four recent methods are considered: DropEdge (Rong et al., 2020),  AdaEdge (Chen et al., 2020a), PairNorm (Zhao & Akoglu, 2020), and BBGDC (Hasanzadeh et al., 2020).
Revised: For addressing the oversmoothing issue, ﬁve methods are considered: DropEdge (Rong et al., 2020), DropICE, AdaEdge (Chen et al., 2020a), PairNorm (Zhao & Akoglu, 2020), and BBGDC (Hasanzadeh et al., 2020), in which DropICE is implemented by removing the inter-class edges derived from V l .

Original: For tackling uncertain graphs, we compare against several Bayesian approaches including BGCN (Zhang et al., 2019), VGCN (Tiao et al., 2019), and G 3 NN (Ma et al., 2019).
Revised: For tackling uncertain graphs, we compare against several Bayesian approaches including BGCN (Zhang et al., 2019), VGCN (Tiao et al., 2019), and G 3 NN (Ma et al., 2019).

Original: For topology optimization, LDS(Franceschi et al., 2019)
Revised: For topology optimization, LDS

Original: , GDC (Klicpera et al., 2019b),and TO-GCN (Yang et al., 2019) are the baselines.
Revised: [Table] (Franceschi et al., 2019), GDC (Klicpera et al., 2019b), TO-GCN (Yang et al., 2019), GRCN (Yu et al., 2020), and IDGL (Chen et al., 2020c) are the baselines.

Original: GMNN (Qu et al., 2019) is also taken as a baseline, as it also employsthe variational EMalgorithm for  node classiﬁcation.
Revised: GMNN (Qu et al., 2019) is also taken as a baseline, as it also employs variational EM for transductive node classiﬁcation.

Original: We conduct node classiﬁcation under two experimental settings, i.e., full-supervised and label-scarce settings.
Revised: We conduct node classiﬁcation under two experimental settings, i.e., full-supervised and label-scarce settings.

Original: The full-supervised setting follows DropEdge (Rong et al., 2020), where each dataset is split into 500 nodes for validation, 1000 nodes for test and the rest for training.
Revised: The full-supervised setting follows DropEdge (Rong et al., 2020), where each dataset is split into 500 nodes for validation, 1000 nodes for test and the rest for training.

Original: The label-scarce setting assigns labels to only a few nodes and selects 500 nodes for validation, while the rest are used for test.
Revised: The label-scarce setting assigns labels to only a few nodes and selects 500 nodes for validation, while the rest are used for test.

Original: Under the label-scarce setting, we compare VEM-GCN with the baselines except for LDS, as LDS always uses the validation set for training, which is unfair for learning with limited training samples.
Revised: Under the label-scarce setting, we compare VEM-GCN with the baselines except for LDS, as LDS always uses the validation set for training, which is unfair for learning with limited training samples.

Revised: DropICE is also omitted since the number of the removed inter-class edges derived from

Revised: V l is very small in the label-scarce setting and thus DropICE only obtains similar performance as the vanilla GCN.

Original: Considering that the classiﬁcation performance is highly inﬂuenced by the split of the dataset, we run all the models with the same 5 random data splits for each evaluation.
Revised: Considering that the classiﬁcation performance is highly inﬂuenced by the split of the dataset (Shchur et al., 2018), we run all the models with the same 5 random data splits for each evaluation.

Original: To further ensure the credibility of the results, we perform 10 random weight initializations for each data split and report the average test accuracy for both experimental settings.
Revised: To further ensure the credibility of the results, we perform 10 random weight initializations for each data split and report the average test accuracy for both experimental settings.

Original: Model Conﬁgurations.
Revised: Model Conﬁgurations.

Original: For  fair comparison, we evaluate all the methods under the same GCN backbone and the same training procedure.
Revised: For a fair comparison, we evaluate all the methods under the same GCN backbone and the same training procedure.

Original: To be concrete, the graph model used in all baselines

Original: [Table]

Original: metrics, the larger the better.

Original: A: Accuracy.

Original: S: Over-smoothness. [Table] and our VEM-GCN ( p θ 1 in the M-step) is a vanilla GCN (Kipf & Welling, 2017) with the number of hidden units set as 32 .
Revised: To be concrete, the graph model used in all baselines and our VEM-GCN ( p θ 1 in the M-step) is a vanilla GCN (Kipf & Welling, 2017) with the number of hidden units set as 32 .

Original: Besides [Table] we train  all the models for each dataset with the same dropout rate of 0 .
Revised: Besides, we train the GCN backbone of all the methods for each dataset with the same dropout rate of 0 .

Original: 5 , the same weight decay, the same learning rate of 0 .
Revised: 5 , the same weight decay, the same learning rate of 0 .

Original: 01 , the same optimizer (Adam (Kingma & Ba, 2015)), the same maximum training epoch of [Table] and the same early stopping strategy based on the validation loss with a patience of 50 epochs.
Revised: 01 , the same optimizer (Adam (Kingma & Ba, 2015)), the same maximum training epoch of 1500 , and the same early stopping strategy based on the validation loss with a patience of 50 epochs (for deeper models with more than 2 layers, we set the patience as 100 epochs).

Revised: Note that IDGL empirically needs more training epochs to converge and we set its maximum training epoch as 10000 with a patience of 1000 epochs.

Original: For q φ in the E-step, the input node embeddings are the attributes averaged over the neighborhood of each node and the network architecture is a four-layer MLP with hidden units of size 128 , 64 , 64 , and 32 , respectively.
Revised: As for q φ in the E-step, the input node embeddings are the attributes averaged over the neighborhood of each node and the network architecture is a four-layer MLP with hidden units of size 128 , 64 , 64 , and 32 , respectively.

Original: Please refer to Appendix D for more details of the  hyperparameter settings for each dataset.
Revised: Please refer to Appendix D for more details of the implementations and hyperparameter settings for each dataset.

Original: 4. R ESULTS AND A NALYSIS
Revised: 4. R ESULTS AND A NALYSIS

Original: Full-supervised Setting.
Revised: Full-supervised Setting.

Original: Table 1 summarizes the classiﬁcation results.
Revised: Table 1 summarizes the classiﬁcation results.

Original: The highest accuracy in each column is highlighted in bold.
Revised: The highest accuracy in each column is highlighted in bold.

Original: Note that the results of BBGDC and LDS on three large graphs (i.e., Pubmed, Amazon Computers, and MS Academic)  are missing due to the out-of-memory error.
Revised: Note that the results of BBGDC and LDS on three large graphs (i.e., Pubmed, Amazon Computers, and MS Academic) and IDGL on MS Academic graph are missing due to the out-of-memory error.

Original: Table 1 demonstrates that none of the baselines outperform the vanilla GCN in all cases, while VEMGCN consistently improves the test accuracy of the GCN backbone by noticeable margins.
Revised: Table 1 demonstrates that none of the baselines outperform the vanilla GCN in all cases, while VEM-GCN consistently improves the test accuracy of the GCN [Table] DropEdge [A / S] [Table]

Original: Speciﬁcally, we ﬁnd the following facts under the full-supervised setting: (1) For tackling over-smoothing,
Revised: [A /

Original: AdaEdge, DropEdge and PairNorm demonstrate limited improvement on several datasets, while
Revised: [Table] VEM [Table] [A / S

Original:  BBGDC  almost collapses for all cases.
Revised: [Table] [Table] backbone by noticeable margins. Speciﬁcally, we ﬁnd the following facts under the full-supervised setting: (1) For tackling over-smoothing, AdaEdge, DropEdge and PairNorm demonstrate limited improvement on several datasets, while BBGDC and DropICE almost collapse for all cases.

Original: (2) LDS, TO-GCN,  and GDC cannot guarantee that their topology optimization could achieve performance gains for the GCN backbone.
Revised: LDS, TO-GCN, GDC, GRCN and IDGL cannot guarantee that their topology optimization could achieve performance gains for the GCN backbone.

Revised: (3) Only adding intra-class edges (TO-GCN) or removing inter-class edges (DropICE) derived from V l might cause topology imbalance between V u and V l .

Revised: The GCN trained on V l with enhanced graph topology would fail on V u with the original graph topology.

Original: (3) Bayesian approaches and GMNN can only achieve comparable performance with the vanilla GCN in almost all cases.
Revised: (4) Bayesian approaches and GMNN can only achieve comparable performance with the vanilla GCN in almost all cases.

Original: Overall, these facts imply that VEM-GCN signiﬁcantly beneﬁts from the large labeled data to generate a clearer topology and achieve better performance.
Revised: Overall, these facts imply that VEM-GCN signiﬁcantly beneﬁts from the large labeled data to generate a clearer topology and achieve better performance.

Original: Label-scarce Setting.
Revised: Label-scarce Setting.

Original: We randomly select 10 labeled nodes per class as the training set and evaluate the performance of VEM-GCN with varying layers.
Revised: We randomly select 10 labeled nodes per class as the training set and evaluate the performance of VEM-GCN with varying layers.

Original: Table 2 shows the test accuracy and the oversmoothness measurements of the learned node embeddings (input node features of the last layer).
Revised: Table 2 shows the test accuracy and the oversmoothness measurements of the learned node embeddings (input node features of the last layer).

Original: [Table]

Original: The metric to measure the over-smoothness is deﬁned in Appendix D.3 and supplementary results  on additional datasets are shown in Appendix E.1.
Revised: The metric to measure the over-smoothness is deﬁned in Appendix D.3 and supplementary results of VEM-GCN on additional datasets are shown in Appendix E.1.

Revised: As can be seen in Table 2, the vanilla

Original: The vanilla GCN severely suffers from the oversmoothing issue, while VEM-GCN can achieve performance gains even with deeper layers.
Revised: GCN severely suffers from the over-smoothing issue, while VEM-GCN can achieve performance gains even with deeper layers (e.g., on the Cora dataset).

Revised: DropEdge and AdaEdge can relieve the over-smoothing issue to some extent, but the performance still decreases drastically when stacking more GCN layers.

Original: The results for over-smoothness measurement indicate that VEM-GCN indeed produces more separable node embeddings across different classes.
Revised: The results of over-smoothness measurements indicate that VEM-GCN indeed produces more separable node embeddings across different classes to address the over-smoothing problem.

Original: We further take Amazon Photo as an example dataset to validate VEM-GCN under different label rates.
Revised: We further take Amazon Photo as an example dataset to validate VEM-GCN under different label rates.

Original: Similar trend as Table 1 can be found in Table 3.
Revised: Similar trend as Table 1 can be found in Table 3.

Original:  Visualization Results.
Revised: Convergence Analysis and Visualization Results.

Original: VEM-GCN leverages the variational EM algorithm for optimization.
Revised: VEM-GCN leverages the variational EM algorithm for optimization.

Original: In this subsection, we analyze the convergence of VEM-GCN.
Revised: In this subsection, we analyze the convergence of VEM-GCN.

Original: Figuredepicts the accuracy improvement curve of p θ 1 during the EM iterations under the full-supervised setting.
Revised: Figuredepicts the accuracy improvement curve of p θ 1 during the EM iterations under the full-supervised setting.

Original: We ﬁnd that VEM-GCN requires only a few iterations to converge.
Revised: We ﬁnd that VEM-GCN requires only a few iterations to converge.

Original: We further take CoraML as an example to give the corresponding visualization results of graph topology optimization.
Revised: We further take CoraML as an example to give the corresponding visualization results of graph topology optimization.

Original: Figure 2 show that the observed graph is very sparse and contains a few intra-class edges, while the optimized graph recover many missing intra-class edges to relieve the over-smoothing problem.
Revised: Figure 2 show that the observed graph is very sparse and contains a few intra-class edges, while the optimized graph recover many missing intra-class edges to relieve the over-smoothing problem.

Original: Note that, although the reﬁned graph is much denser than the observed graph, the parameter p ( 0 .here) in ¯ q φ helps maintain the sparsity of the  adjacency matrix in the training procedure.
Revised: Note that, although the reﬁned graph is much denser than the observed graph, the hyperparameter p ( 0 . 05 here) in ¯ q φ helps maintain the sparsity of the latent adjacency matrix in the training procedure.

Original: Thus, the M-step can still be implemented efﬁciently using sparse-dense matrix multiplications.
Revised: Thus, the M-step can still be implemented efﬁciently using sparse-dense matrix multiplications.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present a novel architecture termed VEM-GCN for addressing the over-smoothing problem in GCNs with graph topology optimization.
Revised: In this paper, we present a novel architecture termed VEM-GCN for addressing the over-smoothing problem in GCNs with graph topology optimization.

Original: By introducing a latent graph parameterized by the assortative-constrained stochastic block model and utilizing the variational EM algorithm to jointly optimize the graph structure and the likelihood function, VEM-GCN outperforms a variety of state-of-the-art methods for tackling over-smoothing, uncertain graphs, and topology optimization in GCNs.
Revised: By introducing a latent graph parameterized by the assortative-constrained stochastic block model and utilizing the variational EM algorithm to jointly optimize the graph structure and the likelihood function, VEM-GCN outperforms a variety of state-of-the-art methods for tackling over-smoothing, uncertain graphs, and topology optimization in GCNs.

Original: For future work, we expect further improvements for the VEM-GCN architecture to deal with more complex graphs such as hypergraphs and heterogeneous graphs.
Revised: For future work, we expect further improvements for the VEM-GCN architecture to deal with more complex graphs such as hypergraphs and heterogeneous graphs

Original: A A LGORITHM
Revised: A A LGORITHM

Original: For a fair comparison, we adopt the vanilla GCN (Kipf & Welling, 2017) as the backbone for all baselines and our proposed VEM-GCN architecture.
Revised: For a fair comparison, we adopt the vanilla GCN (Kipf & Welling, 2017) as the backbone for all baselines and our proposed VEM-GCN architecture.

Original: A two-layer GCN is in the following form:
Revised: A two-layer GCN is in the following form:

Original: [Equation]
Revised: [Equation]

Original: where X is the input node attribute matrix, I N is the identity matrix, ˙
Revised: where X is the input node attribute matrix, I N is the identity matrix, ˙

Original: A =
Revised: A =

Original: A + I N is the adjacency matrix with added self-loops, ˙ D is its corresponding diagonal degree matrix, and θ 1 = { Θ ( 0 ) , Θ ( 1 ) } are the learnable weight parameters.
Revised: A + I N is the adjacency matrix with added self-loops, ˙ D is its corresponding diagonal degree matrix, and θ 1 = { Θ ( 0 ) , Θ ( 1 ) } are the learnable weight parameters.

Original: Algorithm 1 elaborates the proposed VEM-GCN architecture.
Revised: Algorithm 1 describes the proposed VEM-GCN architecture.

Original: B F URTHER D ISCUSSIONS
Revised: B F URTHER D ISCUSSIONS

Original: In addition to recent strategies for tackling over-smoothing issues, we further distinguish VEM-GCN from the SBM-related GCNs and VGCN and GMNN (that introduce variational inference).
Revised: In addition to recent strategies for tackling over-smoothing issues, we further distinguish VEM-GCN from the SBM-related GCNs and VGCN and GMNN (that introduce variational inference)

Original: Comparison with SBM-related GCNs.
Revised: Comparison with SBM-related GCNs.

Original: Stochastic block model (SBM) has been combined with
Revised: Stochastic block model (SBM) has been combined with

Original: GCNs in several recent works (i.e., BGCN (Zhang et al., 2019) and G 3 NN (Ma et al., 2019)).
Revised: GCNs in several recent works (i.e., BGCN (Zhang et al., 2019) and G 3 NN (Ma et al., 2019)).

Original: However, these architectures are totally different from VEM-GCN in motivations, objective functions and training methods.
Revised: However, these architectures are totally different from VEM-GCN in motivations, objective functions and training methods.

Original: BGCN (Zhang et al., 2019) jointly infers the node labels and the parameters of SBM using only A obs , which ignores the dependence of the graph on X and Y l .
Revised: BGCN (Zhang et al., 2019) jointly infers the node labels and the parameters of SBM using only A obs , which ignores the dependence of the graph on X and Y l .

Original: Then the adjacency matrices sampled from the inferred SBM are used to train the GCN.
Revised: Then the adjacency matrices sampled from the inferred SBM are used to train the GCN.

Original: Different from VEMGCN, BGCN neither explicitly promotes intra-class connection nor demotes inter-class interaction.
Revised: Different from VEMGCN, BGCN neither explicitly promotes intra-class connection nor demotes inter-class interaction.

Original: It only achieves robustness under certain conditions such as adversarial attacks, beneﬁting from the uncertainty brought by the inferred SBM.
Revised: It only achieves robustness under certain conditions such as adversarial attacks, beneﬁting from the uncertainty brought by the inferred SBM.

Original: G 3 NN is a ﬂexible generative model, where the graph generated by SBM is based on the predictions of an additional MLP learned from only X and Y l .
Revised: G 3 NN is a ﬂexible generative model, where the graph generated by SBM is based on the predictions of an additional MLP learned from only X and Y l .

Original: The predictions for the unlabeled nodes are still based on G obs (i.e., the input adjacency matrix of the
Revised: The predictions for the unlabeled nodes are still based on G obs (i.e., the input adjacency matrix of the

Original: GCN is still A obs ).
Revised: GCN is still A obs ).

Original: By contrast, VEM-GCN aims at addressing the over-smoothing issue with topology optimization.
Revised: By contrast, VEM-GCN aims at addressing the over-smoothing issue with topology optimization.

Original: In VEM-GCN, the M-step trains a GCN to obtain the predictions of the unlabeled nodes based on A latent , X , and Y l .
Revised: In VEM-GCN, the M-step trains a GCN to obtain the predictions of the unlabeled nodes based on A latent , X , and Y l .

Original: We then estimate the posterior distribution on A latent based on Y l and the predictions for the unlabeled nodes under the SBM assumption.
Revised: We then estimate the posterior distribution on A latent based on Y l and the predictions for the unlabeled nodes under the SBM assumption.

Original: Subsequently, the E-step optimizes the graph topology by training another auxiliary neural network with node embeddings as input to approximate the posterior distribution of A latent .
Revised: Subsequently, the E-step optimizes the graph topology by training another auxiliary neural network with node embeddings as input to approximate the posterior distribution of A latent .

Original: The E-step and M-step are optimized in an alternating fashion to improve each other.
Revised: The E-step and M-step are optimized in an alternating fashion to improve each other.

Original: VEM-GCN vs. VGCN.
Revised: VEM-GCN vs. VGCN.

Original: VGCN (Tiao et al., 2019) also introduces a latent graph A latent and optimizes L ELBO in Eq.
Revised: VGCN (Tiao et al., 2019) also introduces a latent graph A latent and optimizes L ELBO in Eq.

Original: However, it directly optimizes the ELBO in a VAE (Kingma & Welling, 2014) fashion and the posterior distribution of A latent is set to approximate the pre-deﬁned graph priors p ( a prior ij = 1)
Revised: However, it directly optimizes the ELBO in a VAE (Kingma & Welling, 2014) fashion and the posterior distribution of A latent is set to approximate the pre-deﬁned graph priors p ( a prior ij = 1)

Original: =
Revised: =

Original: ρ 1 a obs ij + ρ 2 (1 − a obs ij ) with 0 < ρ 1 , ρ 2 < 1 using the re-parameterization trick.
Revised: ρ 1 a obs ij + ρ 2 (1 − a obs ij ) with 0 < ρ 1 , ρ 2 < 1 using the re-parameterization trick.

Original: VGCN is to achieve robustness under fake link attacks and only shows comparable performance with GCN under the standard transductive learning setting (i.e., inferring Y u based on  G obs ).
Revised: VGCN is to achieve robustness under fake link attacks and only shows comparable performance with GCN under the standard transductive learning setting (i.e., inferring Y u based on the original G obs ).

Original: By contrast, VEM-GCN does not introduce priors over graphs.
Revised: By contrast, VEM-GCN does not introduce priors over graphs.

Original: We optimize the graph topology by explicitly enhancing intra-class connection and suppressing inter-class interaction using SBM and variational EM to relieve the over-smoothing issue.
Revised: We optimize the graph topology by explicitly enhancing intra-class connection and suppressing inter-class interaction using SBM and variational EM to relieve the over-smoothing issue.

Original: VEM-GCN vs. GMNN.
Revised: VEM-GCN vs. GMNN.

Original: Graph Markov Neural Network (GMNN) (Qu et al., 2019) also employs variational EM for node classiﬁcation, but it is totally different from our method in motivations and objective functions.
Revised: Graph Markov Neural Network (GMNN) (Qu et al., 2019) also employs variational EM for node classiﬁcation, but it is totally different from our method in motivations and objective functions.

Original: GMNN focuses on modeling the joint distribution of object (node) labels.
Revised: GMNN focuses on modeling the joint distribution of object (node) labels.

Original: Therefore, GMNN views Y u as latent variables and optimizes the following ELBO:
Revised: Therefore, GMNN views Y u as latent variables and optimizes the following ELBO:

Original: [Equation]
Revised: [Equation]

Original: In the E-step, GMNN parameterizes q φ ( Y u | X ) with a GCN and q φ ( Y u | X ) is optimized to approximate the posterior distribution p θ ( Y u | Y l , X ) .
Revised: In the E-step, GMNN parameterizes q φ ( Y u | X ) with a GCN and q φ ( Y u | X ) is optimized to approximate the posterior distribution p θ ( Y u | Y l , X ) .

Original: In the M-step, GMNN utilizes another GCN to model the conditional distribution p θ ( y i | y NB ( i ) , X ) for each node v i ∈ V (NB ( i ) is the neighbor[Table] set of node v i ) with a conditional random ﬁeld and maximizes the corresponding likelihood.
Revised: In the M-step, GMNN utilizes another GCN to model the conditional distribution p θ ( y i | y NB ( i ) , X ) for each node v i ∈ V (NB ( i ) is the neighbor set of node v i ) with a conditional random ﬁeld and maximizes the corresponding likelihood.

Original: On the contrary, VEM-GCN is proposed to relieve the over-smoothing issue.
Revised: On the contrary, VEM-GCN is proposed to relieve the over-smoothing issue.

Original: VEM-GCN optimizes the  latent graph to approximate its posterior distribution based on SBM in the E-step, and trains a GCN based on the latent graph in the M-step.
Revised: VEM-GCN optimizes the [Table] latent graph to approximate its posterior distribution based on SBM in the E-step, and trains a GCN based on the latent graph in the M-step.

Original: C D ATASET S TATISTICS
Revised: C D ATASET S TATISTICS

Original: We utilize seven node classiﬁcation benchmarks in this paper, including four citation networks (i.e.,
Revised: We utilize seven node classiﬁcation benchmarks in this paper, including four citation networks (i.e.,

Original: Citeseer, Pubmed, Cora, and Cora-ML), two Amazon co-purchase graphs (i.e., Amazon Photo and
Revised: Citeseer, Pubmed, Cora, and Cora-ML), two Amazon co-purchase graphs (i.e., Amazon Photo and

Original: Amazon Computers), and one Microsoft Academic graph, as summarized below.
Revised: Amazon Computers), and one Microsoft Academic graph, as summarized below.

Original: • Citation Networks .
Revised: • Citation Networks .

Original: Cora, Citeseer, Pubmed can be downloaded from the ofﬁcial source code of GCN (Kipf & Welling, 2017) publicly available at https://github.com/ tkipf/gcn/tree/master/gcn/data , and Cora-ML can be downloaded from the source code of (A)PPNP (Klicpera et al., 2019a) publicly available at https:// github.com/klicperajo/ppnp/tree/master/ppnp/data .
Revised: Cora, Citeseer, Pubmed can be downloaded from the ofﬁcial source code of GCN (Kipf & Welling, 2017) publicly available at https://github.com/ tkipf/gcn/tree/master/gcn/data , and Cora-ML can be downloaded from the source code of (A)PPNP (Klicpera et al., 2019a) publicly available at https:// github.com/klicperajo/ppnp/tree/master/ppnp/data .

Original: • Amazon Co-purchase Graph .
Revised: • Amazon Co-purchase Graph .

Original: The Amazon Photo and Amazon Computers datasets from the Amazon co-purchase graph can be publicly downloaded from https://github.
Revised: The Amazon Photo and Amazon Computers datasets from the Amazon co-purchase graph can be publicly downloaded from https://github.

Original: com/shchur/gnn-benchmark/tree/master/data (Shchur et al., 2018).
Revised: com/shchur/gnn-benchmark/tree/master/data (Shchur et al., 2018).

Original: • Microsoft Academic Graph .
Revised: • Microsoft Academic Graph .

Original: The MS Academic graph can be downloaded from the source code of (A)PPNP (Klicpera et al., 2019a) publicly available at https://github.com/ klicperajo/ppnp/tree/master/ppnp/data .
Revised: The MS Academic graph can be downloaded from the source code of (A)PPNP (Klicpera et al., 2019a) publicly available at https://github.com/ klicperajo/ppnp/tree /master/ppnp/data .

Original: An overview of the dataset statistics is listed in Table 4.
Revised: An overview of the dataset statistics is listed in Table 4.

Original: Note that for these open datasets, three (Cora, Citeseer, Pubmed) are given in the form of undirected graphs, while four (Cora-ML, Amazon
Revised: Note that for these open datasets, three (Cora, Citeseer, Pubmed) are given in the form of undirected graphs, while four (Cora-ML, Amazon

Original: Photo, Amazon Computers, MS Academic) are directed graphs.
Revised: Photo, Amazon Computers, MS Academic) are directed graphs.

Original: GCN treats all these datasets as undirected graphs (i.e., a (cid:48) ij = [ a ij + a ji > 0] , where [ · ] denotes Iverson bracket).
Revised: GCN treats all these datasets as undirected graphs (i.e., a (cid:48) ij = [ a ij + a ji > 0] , where [ · ] denotes Iverson bracket).

Original: D F URTHER E XPERIMENTAL D ETAILS D. I MPLEMENTATIONS
Revised: D F URTHER E XPERIMENTAL D ETAILS D. I MPLEMENTATIONS

Original: The implementation of VEM-GCN consists of two alternating steps in each iteration, including a variational E-step and an M-step.
Revised: The implementation of VEM-GCN consists of two alternating steps in each iteration, including a variational E-step and an M-step.

Original: In the variational E-step, a simple four-layer MLP is implemented for q φ , where the numbers of neuron units of each layer are 128 , 64 , 64 , and 32 , respectively.
Revised: In the variational E-step, a simple four-layer MLP is implemented for q φ , where the numbers of neuron units of each layer are 128 , 64 , 64 , and 32 , respectively.

Original: We use tanh as the nonlinear activation function for the hidden layers.
Revised: We use tanh as the nonlinear activation function for the hidden layers.

Original: In the M-step, p θ 1 is a vanilla GCN with the number of hidden units set as 32 , and we use the ofﬁcial source code from https://github.com/tkipf/gcn/tree/master/gcn .
Revised: In the M-step, p θ 1 is a vanilla GCN with the number of hidden units set as 32 , and we use the ofﬁcial source code from https://github.com/tkipf/gcn/tree/master/gcn .

Original: All the baselines and our VEMGCN architecture are trained on a single NVIDIA GTX 1080 Ti GPU with 11GB memory.
Revised: All the baselines and our VEMGCN architecture are trained on a single NVIDIA GTX 1080 Ti GPU with 11GB memory.

Original: We just utilize the raw node attributes X as the input to q φ .
Revised: We just utilize the raw node attributes X as the input to q φ .

Original: Note that we can also support any other desirable network embedding method and these experiments is left for the future work.
Revised: Note that we can also support any other desirable network embedding method and these experiments is left for the future work.

Original: Considering the fact that the bag-of-words representations of X is often noisy, we average the features of each node over its neighborhoods to smooth the input signal.
Revised: Considering the fact that the bag-of-words representations of X is often noisy, we average the features of each node over its neighborhoods to smooth the input signal.

Original: Let ˆ A row =
Revised: Let ˆ A row =

Original: ( D + γ I N ) − 1 ( A + γ
Revised: ( D + γ I N ) − 1 ( A + γ

Original: I N ) denote the “self-enhanced” adjacency matrix with row normalization (we use γ = 1 . 5 in this paper), and [ A (cid:107) B ] be the concatenation of matrices A and B along the last dimension.
Revised: I N ) denote the “self-enhanced” adjacency matrix with row normalization (we use γ = 1 . 5 in this paper), and [ A (cid:107) B ] be the concatenation of matrices A and B along the last dimension.

Original: Consequently, we summarize the speciﬁc input to q φ for all the datasets as below.
Revised: Consequently, we summarize the speciﬁc input to q φ for all the datasets as below.

Original: • For Cora, Citeseer, and MS Academic, we use X (cid:48)
Revised: • For Cora, Citeseer, and MS Academic, we use X (cid:48)

Original: =
Revised: =

Original: ˆ A row X as the input of q φ .
Revised: ˆ A row X as the input of q φ .

Original: • For Pubmed and Amazon Photo, we use X (cid:48)
Revised: • For Pubmed and Amazon Photo, we use X (cid:48)

Original: = (cid:104) X (cid:107) ˆ A row X (cid:107) ˆ
Revised: = (cid:104) X (cid:107) ˆ A row X (cid:107) ˆ

Original: A 2 row X (cid:105) as the input of q φ .
Revised: A 2 row X (cid:105) as the input of q φ .

Original: • For Cora-ML and Amazon Computers, we use X (cid:48)
Revised: • For Cora-ML and Amazon Computers, we use X (cid:48)

Original: = (cid:104) X (cid:107)
Revised: = (cid:104) X (cid:107)

Original: ˆ A row X (cid:105)
Revised: ˆ A row X (cid:105)

Original: as the input of q φ .
Revised: as the input of q φ .

Original: For the sampling in the E-step, we ﬁnd that it is not always stable to draw the sampled labels ˆ
Revised: For the sampling in the E-step, we ﬁnd that it is not always stable to draw the sampled labels ˆ

Original: Y u ∼ p θ ( Y u | Y l , G obs ) .
Revised: Y u ∼ p θ ( Y u | Y l , G obs ) .

Original: To alleviate this problem, we maintain ˆ Y u ← argmax y
Revised: To alleviate this problem, we maintain ˆ Y u ← argmax y

Original: ( p θ ( Y u | Y l , G obs )) with probability p e , and sample
Revised: ( p θ ( Y u | Y l , G obs )) with probability p e , and sample

Original: ˆ Y u ∼ p θ ( Y u | Y l , G obs ) with probability (1 − p e ) to improve the performance in practice.
Revised: ˆ Y u ∼ p θ ( Y u | Y l , G obs ) with probability (1 − p e ) to improve the performance in practice.

Revised: For training of the E-step, we utilize SGD with momentum (of 0 . 99 ).

Original: In the test procedure, we perform inference in two ways: (1) keep p in Eq. 9 the same as the training procedure (commonly p < 1 ), sample S ( S > 1) adjacency matrices, and predict the classes for the unlabeled nodes according to Eq. 11; (2) set p = 1 (i.e., the latent adjacency matrix in now deterministic) and S = 1 .
Revised: In the test procedure, we perform inference in two ways: (1) keep p in Eq. 9 the same as the training procedure (commonly p < 1 ), sample S ( S > 1) adjacency matrices, and predict the classes for the unlabeled nodes according to Eq. 11; (2) set p = 1 (i.e., the latent adjacency matrix is now deterministic) and S = 1 .

Original: We report the best test accuracy obtained using these two sampling methods on each dataset and ﬁnd that (2) almost always performs better.
Revised: We report the best test accuracy obtained using these two sampling methods on each dataset and ﬁnd that (2) almost always performs better.

Original: D. H YPERPARAMETER S ETTINGS
Revised: D. H YPERPARAMETER S ETTINGS

Original: Table 5 summarizes the hyperparameters adopted for the full-supervised setting on the seven node classiﬁcation benchmarks.
Revised: Table 5 summarizes the hyperparameters adopted for the full-supervised setting on the seven benchmark datasets.

Original: For the label-scarce setting, the hyperparameters are the same, except for ε 1 that needed to be carefully tuned in the search space { 0 .  99 , 0 . 995 , 0 . 999 , 0 .
Revised: For the label-scarce setting, the hyperparameters are the same, except for ε 1 and εthat need to be carefully tuned in the search space: ε 1 ∈ { 0 . 95 , 0 . 99 , 0 . 995 , 0 . 999 , 0 .

Original: 9995 , 0 . 9999 } [-for each data split. a obs ij = 0) = 
 and λ ( a obs ij = 1)-]
Revised: 9995 , 0 . 9999 } [-for each data split. a obs ij = 0) = 
 and λ ( a obs ij = 1)-], ε 2 ∈ { 0 . 001 , 0 . 005 , 0 . 01 , 0 . 05 ,

Original: = λ .
Revised: [Table]

Original: lr: Learning rate.

Original: [Table]
Revised: [Table]

Original: D. M ETRIC FOR M EASURING O VER - SMOOTHNESS
Revised: D. M ETRIC FOR M EASURING O VER - SMOOTHNESS

Original: To address the over-smoothing issue, one would prefer to reduce the intra-class distance to make node features in the same class similar, and increase the inter-class distance to produce distinguishable representations for nodes in different classes.
Revised: To address the over-smoothing issue, one would prefer to reduce the intra-class distance to make node features in the same class similar, and increase the inter-class distance to produce distinguishable representations for nodes in different classes.

Original: Hence, we use the ratio of average inter-class distance to average intra-class distance (Euclidean distance of input node features in the last layer) to measure the over-smoothness.
Revised: Hence, we use the ratio of average inter-class distance to average intra-class distance (Euclidean distance of input node features in the last layer) to measure the over-smoothness.

Original: Formally, given the learned node embeddings H = { h i } Ni =1 (the input node features of the last layer), we ﬁrst calculate the distance matrix D =
Revised: Formally, given the learned node embeddings H = { h i } Ni =1 (the input node features of the last layer), we ﬁrst calculate the distance matrix D =

Original: [ d ij ] ∈ R N × N with each entry deﬁned as
Revised: [ d ij ] ∈ R N × N with each entry deﬁned as

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where (cid:107) · (cid:107) 2 denotes Euclidean norm.
Revised: 15) where (cid:107) · (cid:107) 2 denotes Euclidean norm.

Original: Next, we deﬁne the inter-class mask matrix and intra-class mask matrix as follows:
Revised: Next, we deﬁne the inter-class mask matrix and intra-class mask matrix as follows:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 17) where
Revised: (17) where

Original: ˜ A =
Revised: ˜ A =

Original: YY (cid:62) is the optimal graph and 1 N × N is a matrix of size N × N with all entries set to 1 .
Revised: YY (cid:62) is the optimal graph and 1 N × N is a matrix of size N × N with all entries set to 1 .

Original: Then we can obtain the inter-class distance matrix
Revised: Then we can obtain the inter-class distance matrix

Original: D inter =
Revised: D inter =

Original: [ d interij ] ∈ R N × N and the intraclass distance matrix D intra
Revised: [ d interij ] ∈ R N × N and the intraclass distance matrix D intra

Original: =
Revised: =

Original: [ d intra ij ] ∈ R N × N by element-wise multiplication D with the mask matrices:
Revised: [ d intra ij ] ∈ R N × N by element-wise multiplication D with the mask matrices:

Original: [Equation]
Revised: [Equation]

Original: where ◦ denotes element-wise multiplication.
Revised: where ◦ denotes element-wise multiplication.

Original: Next we get the average inter-class distance AD inter and the average intra-class distance AD intra , with which we measure the over-smoothness as their ratio R :
Revised: Next we get the average inter-class distance AD inter and the average intra-class distance AD intra , with which we measure the over-smoothness as their ratio R :

Original: [Equation]
Revised: [Equation]

Original: where 1 ( x ) = 1 if x > 0 otherwise 0 .
Revised: where 1 ( x ) = 1 if x > 0 otherwise 0 .

Original: E F URTHER E XPERIMENTAL R ESULTS E. C LASSIFICATION R ESULTS UNDER THE L ABEL -S CARCE S ETTING
Revised: E F URTHER E XPERIMENTAL R ESULTS E. C LASSIFICATION R ESULTS UNDER THE L ABEL -S CARCE S ETTING

Original: Supplementary experiments for Table 2 are shown in Table
Revised: Supplementary experiments for Table 2 are shown in Table

Original: [Table]
Revised: [Table]

Original: V ISUALIZATION R ESULTS
Revised: V ISUALIZATION R ESULTS

Original: Figure 2 demonstrates the topology optimization results on the Cora-ML dataset under the fullsupervised setting.
Revised: Figure 2 demonstrates the topology optimization results on the Cora-ML dataset under the fullsupervised setting.

Original: For the lable-scarce setting, the visualization results are shown in Figure 3.
Revised: For the lable-scarce setting, the visualization results are shown in Figure 3.

Original: We further take Cora-ML as an example dataset to provide t-SNE (Maaten & Hinton, 2008) visualizations of the learned node embeddings (input node features of the last layer) extracted by the vanilla GCN and our proposed VEM-GCN with varying layers under the label-scarce setting (labeled nodes per class).
Revised: We further take Cora-ML as an example dataset to provide t-SNE (Maaten & Hinton, 2008) visualizations of the learned node embeddings (input node features of the last layer) extracted by the vanilla GCN and our proposed VEM-GCN with varying layers under the label-scarce setting (labeled nodes per class).

Original: The results are shown in Figures 4 and 5.
Revised: The results are shown in Figures 4 and 5.

Original: As can be seen, VEM-GCN indeed generates more separable node embeddings for nodes in different classes (colors) for classiﬁcation.
Revised: As can be seen, VEM-GCN indeed generates more separable node embeddings for nodes in different classes (colors) for classiﬁcation.

Original: In particular, a ten-layer vanilla GCN severely suffers from the over-smoothing issue where node features in different classes are over-mixed and thus indistinguishable, while our VEM-GCN architecture with a ten-layer GCN as the backbone still achieves comparable performance with a two-layer model.
Revised: In particular, a ten-layer vanilla GCN severely suffers from the over-smoothing issue where node features in different classes are over-mixed and thus indistinguishable, while our VEM-GCN architecture with a ten-layer GCN as the backbone still achieves comparable performance with a two-layer model.

Original: E. C OMPLEXITY A NALYSIS
Revised: E. C OMPLEXITY A NALYSIS

Original: VEM-GCN is very ﬂexible and general.
Revised: VEM-GCN is very ﬂexible and general.

Original: There is no constraint on the design of the two neural networks q φ and p θ .
Revised: There is no constraint on the design of the two neural networks q φ and p θ .

Original: Therefore, the architecture can be combined with arbitrary GCN models andnode
Revised: Therefore, the architecture can be combined with arbitrary GCN models and

Original: [Table]
Revised: [Table]

Original:  embedding methods.
Revised: node embedding methods.

Original: Moreover, we generalize some existing state-of-the-art strategies for tackling the over-smoothing problem (i e., DropEdge and AdaEdge).
Revised: Moreover, we generalize some existing state-of-the-art strategies for tackling the over-smoothing problem (i e., DropEdge and AdaEdge).

Original: In comparison to the vanillaGCN,
Revised: In comparison to the vanilla

Original:  VEM-GCN achieves these beneﬁts with topology optimization at the cost of efﬁciency.
Revised: GCN, VEM-GCN achieves these beneﬁts with topology optimization at the cost of efﬁciency.

Original: As elaborated in Sections 3.2 and 3.3, the M-step is a traditional training procedure for optimizing the GCNs, while the E-step introduces an extra procedure that trains a MLP for graph structure optimization.
Revised: As illustrated in Sections 3.2 and 3.3, the M-step is a traditional training procedure for optimizing the GCNs. Although A latent recovers more intra-class edges than A obs , the parameter p in Eq.

Revised: 9 helps maintain the sparsity of the latent graph in each step of the training procedure.

Revised: Thus the M-step shares almost the same complexity as GCN.

Revised: The E-step introduces an extra procedure that trains a

Original:  However, to address the over-smoothing issue at the core, we argue thatgraph topology optimization is necessary.
Revised: MLP for graph structure optimization. However, to address the over-smoothing issue at the core, we argue that topology optimization is necessary.

Original: Actually, efﬁciency issue is a common problem for some Bayesian approaches and topology optimization methods.
Revised: Actually, efﬁciency issue is a common problem for some Bayesian approaches and topology optimization methods.

Original: Despite decreased efﬁciency compared with the vanilla GCN, VEM-GCN optimizes A latent with amortized variational inference (i.e., training a MLP shared by all the node pairs with mini-batch SGD in the E-step), which is faster than BGCN (Zhang et al., 2019)
Revised: Despite decreased efﬁciency compared with the vanilla GCN, VEM-GCN optimizes A latent with amortized variational inference (i.e., training a MLP shared by all the node pairs with mini-batch SGD in the E-step), which is faster than BGCN (Zhang et al., 2019)

Original: (Bayesian method) and LDS (Franceschi et al., 2019)
Revised: (Bayesian method) and LDS (Franceschi et al., 2019)

Original: (topology optimization) and is scalable for large-scale graphs.
Revised: (topology optimization) and is scalable for large-scale graphs.

Original: For training on the Amazon Photo dataset with a NVIDIA GTX 1080 Ti GPU, VEM-GCN is about 3x faster than LDS and 4x faster than BGCN.
Revised: For training on the Amazon Photo dataset with a NVIDIA GTX 1080 Ti GPU, VEM-GCN is about 3 × faster than LDS and 4 × faster than BGCN.

Original: E. VEM-GCNII
Revised: E. VEM-GCNII

Original: As mentioned above, our VEM-GCN architecture is ﬂexible and general.
Revised: As mentioned above, our VEM-GCN architecture is ﬂexible and general.

Original: In the E-step, the neural network can support arbitrary desirable node embeddings and the GCN in the M-step can be substituted with any graph models.
Revised: In the E-step, the neural network can support arbitrary desirable node embeddings and the GCN in the M-step can be substituted with any graph models.

Original: This subsection further veriﬁes the effectiveness of our proposed method by trying different models for p θ 1 ( Y l | A latent , G obs ) .
Revised: This subsection further veriﬁes the effectiveness of our proposed method by trying different models for p θ 1 ( Y l | A latent , G obs ) .

Original: We also apply the same node embeddings as illustrated in Appendix D. Trying more effective node embeddings is not the focus of this paper and is left for the future work.
Revised: We also apply the same node embeddings as illustrated in Appendix D. Trying more effective node embeddings is not the focus of this paper and is left for the future work.

Original: Recently, Chen et al.
Revised: Recently, Chen et al.

Original: (2020b) proposed a simple and deep GCN model termed GCNII to address the over-smoothing problem.
Revised: (2020b) proposed a simple and deep GCN model termed GCNII to address the over-smoothing issue.

Original: GCNII improves the vanilla GCN via Initial residual and Identity mapping:
Revised: GCNII improves the vanilla GCN via Initial residual and Identity mapping:

Original: [Equation]
Revised: [Equation]

Original: ) where H (0) =
Revised: where H (0) =

Original: XW (0) is the output of the ﬁrst layer (a fully connected layer), ˜ P
Revised: XW (0) is the output of the ﬁrst layer (a fully connected layer), ˜ P

Original: = ˙
Revised: = ˙

Original: D − 1 2 ˙
Revised: D − 12 ˙

Original: A ˙ D − 1 2 is the normalized adjacency matrix, α ( l ) and β ( l ) are two hyperparameters.
Revised: A ˙ D − 12 is the normalized adjacency matrix, α ( l ) and β ( l ) are two hyperparameters.

Original: We utilize GCNII as the backbone that results in the VEM-GCNII architecture.
Revised: We utilize GCNII as the backbone that results in the VEM-GCNII architecture.

Original: We use the ofﬁcial source code from https://github.com/chennnM/GCNII and employ the hyperparameter settings reported in (Chen et al., 2020b) that achieve the best performance on the three citation networks (i.e., Cora, Citeseer, and Pubmed) under the label-scarce setting.
Revised: We use the ofﬁcial source code from https://github.com/chennnM/GCNII and employ the hyperparameter settings reported in (Chen et al., 2020b) that achieve the best performance on the three citation networks (i.e., Cora, Citeseer, and Pubmed) under the label-scarce setting.

Original: For the other four datasets, we roughly tuned the hyperparameters but found that GCNII does not outperform the vanilla GCN.
Revised: For the other four datasets, we roughly tuned the hyperparameters but found that GCNII does not outperform the vanilla GCN.

Original: Thus, we only evaluate GCNII and VEM-GCNII on Cora, Citeseer, and Pubmed with 10 labeled nodes per class as the training set.
Revised: Thus, we only evaluate GCNII and VEM-GCNII on Cora, Citeseer, and Pubmed with 10 labeled nodes per class as the training set.

Original: Experimental results are shown in Table 7.
Revised: Experimental results are shown in Table 7.

Revised: As can be seen,

Revised: VEM-GCNII consistently boosts the performance of GCNII, further verifying the effectiveness and ﬂexibility of our proposed architecture.