Original: L EARNING G UARANTEES FOR G RAPH C ONVOLUTIONAL N ETWORKS ON THE S TOCHASTIC B LOCK M
Revised: L EARNING G UARANTEES FOR G RAPH C ONVOLUTIONAL N ETWORKS ON THE S TOCHASTIC B LOCK M

Original: ODEL
Revised: ODEL

Original: A BSTRACT
Revised: A BSTRACT

Original: An abundance of neural network models and algorithms for diverse tasks on graphs have been developed in the past five years.
Revised: An abundance of neural network models and algorithms for diverse tasks on graphs have been developed in the past five years.

Original: However, very few provable guarantees have been available for the performance of graph neural network models.
Revised: However, very few provable guarantees have been available for the performance of graph neural network models.

Original: This state of affairs is in contrast with the steady progress on the theoretical underpinnings of traditional dense and convolutional neural networks.
Revised: This state of affairs is in contrast with the steady progress on the theoretical underpinnings of traditional dense and convolutional neural networks.

Original: In this paper we present the first provable guarantees for one of the best-studied families of graph neural network models, Graph Convolutional Networks (GCNs), for semisupervised community detection tasks.
Revised: In this paper we present the first provable guarantees for one of the best-studied families of graph neural network models, Graph Convolutional Networks (GCNs), for semisupervised community detection tasks.

Original: We show that with high probability over the initialization and training data, a GCN will efficiently learn to detect communities on graphs drawn from a stochastic block model.
Revised: We show that with high probability over the initialization and training data, a GCN will efficiently learn to detect communities on graphs drawn from a stochastic block model.

Original: Our proof relies on a fine-grained analysis of the training dynamics in order to overcome the complexity of a nonconvex optimization landscape with many poorly-performing local minima.
Revised: Our proof relies on a fine-grained analysis of the training dynamics in order to overcome the complexity of a nonconvex optimization landscape with many poorly-performing local minima.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There is presently a large gap between what can be accomplished in practice using deep learning, and what can be satisfactorily explained and predicted by the theory of deep learning.
Revised: There is presently a large gap between what can be accomplished in practice using deep learning, and what can be satisfactorily explained and predicted by the theory of deep learning.

Original: Nevertheless, the past several years have seen substantial developments in the theory of deep learning (Ge et al., 2017;
Revised: Nevertheless, the past several years have seen substantial developments in the theory of deep learning (Ge et al., 2017;

Original: Brutzkus & Globerson, 2017; Zhang et al., 2019a; Goel et al., 2020; Chen et al., 2020a).
Revised: Brutzkus & Globerson, 2017; Zhang et al., 2019a; Goel et al., 2020; Chen et al., 2020a).

Original: One factor contributing to the gap between the theory and practice of traditional NNs is that realworld data sets tend to have complex structure that is difficult to capture with formal definitions.
Revised: One factor contributing to the gap between the theory and practice of traditional NNs is that realworld data sets tend to have complex structure that is difficult to capture with formal definitions.

Original: For example, popular image classification models are capable of memorizing arbitrary data (Zhang et al., 2016) and yet they exhibit astonishing generalization performance on accurately-labeled natural images.
Revised: For example, popular image classification models are capable of memorizing arbitrary data (Zhang et al., 2016) and yet they exhibit astonishing generalization performance on accurately-labeled natural images.

Original: Hence, any rigorous proof of the observed generalization performance of deep learning models on image classification tasks will necessarily require assumptions about the data that are sharp enough to separate random inputs from natural images.
Revised: Hence, any rigorous proof of the observed generalization performance of deep learning models on image classification tasks will necessarily require assumptions about the data that are sharp enough to separate random inputs from natural images.

Original: Because of the difficulty of giving an adequate characterization of real-world data, much of the recent progress in deep learning theory has instead focused on proving results using very simple (e.g. Gaussian) input distributions or in distribution-free settings (Ge et al., 2017; Brutzkus & Globerson, 2017; Zhang et al., 2019a; Vempala & Wilmes, 2019).
Revised: Because of the difficulty of giving an adequate characterization of real-world data, much of the recent progress in deep learning theory has instead focused on proving results using very simple (e.g. Gaussian) input distributions or in distribution-free settings (Ge et al., 2017; Brutzkus & Globerson, 2017; Zhang et al., 2019a; Vempala & Wilmes, 2019).

Original: Compared to traditional feed-forward (dense, convolutional, etc.) NNs, the theory of graph neural networks (GNNs) is still in its infancy.
Revised: Compared to traditional feed-forward (dense, convolutional, etc.) NNs, the theory of graph neural networks (GNNs) is still in its infancy.

Original: On the other hand, it appears substantially easier to give plausible descriptions of the combinatorial structure of real-world graph data sets than, e.g., to characterize the distribution of natural images (Drobyshevskiy & Turdakov, 2019).
Revised: On the other hand, it appears substantially easier to give plausible descriptions of the combinatorial structure of real-world graph data sets than, e.g., to characterize the distribution of natural images (Drobyshevskiy & Turdakov, 2019).

Original: We therefore believe that GNNs offer a natural setting for developing provable guarantees that are able to capture the power of deep learning on real-world datasets.
Revised: We therefore believe that GNNs offer a natural setting for developing provable guarantees that are able to capture the power of deep learning on real-world datasets.

Original: In this paper, we contribute to that goal by giving the first rigorous guarantees of efficient semi-supervised learning of stochastic block models via a
Revised: In this paper, we contribute to that goal by giving the first rigorous guarantees of efficient semi-supervised learning of stochastic block models via a

Original: GNN.
Revised: GNN.

Original: 1. G RAPH NEURAL NETWORKS
Revised: 1. G RAPH NEURAL NETWORKS

Original: Many natural datasets for diverse machine learning problems have a graph structure, including social networks, molecular structures, and transit networks.
Revised: Many natural datasets for diverse machine learning problems have a graph structure, including social networks, molecular structures, and transit networks.

Original: In order to efficiently exploit such combinatorial structure, a variety of GNN models have been proposed, tuned for different kinds of tasks.
Revised: In order to efficiently exploit such combinatorial structure, a variety of GNN models have been proposed, tuned for different kinds of tasks.

Original: A number of taxonomies of GNN models have been proposed (Zhou et al., 2018; Wu et al., 2021); one of the most essential differences between different GNN models is whether they are meant to label the graph as a whole, or to label individual components of the graph, particularly vertices.
Revised: A number of taxonomies of GNN models have been proposed (Zhou et al., 2018; Wu et al., 2021); one of the most essential differences between different GNN models is whether they are meant to label the graph as a whole, or to label individual components of the graph, particularly vertices.

Original: From a theoretical perspective, the best understood tasks for GNNs concern labeling the graph as a whole, for example for the task of classifying a graph by its isomorphism type (Sato, 2020).
Revised: From a theoretical perspective, the best understood tasks for GNNs concern labeling the graph as a whole, for example for the task of classifying a graph by its isomorphism type (Sato, 2020).

Original: In particular, it has been established that many GNN models are of comparable power to various versions of the Weisfeiler-Leman hierarchy 1 (Xu et al., 2018; Morris et al., 2019).
Revised: In particular, it has been established that many GNN models are of comparable power to various versions of the Weisfeiler-Leman hierarchy 1 (Xu et al., 2018; Morris et al., 2019).

Original: Some progress has also been made on the theory of GNNs for vertex-labeling tasks.
Revised: Some progress has also been made on the theory of GNNs for vertex-labeling tasks.

Original: Recent works by
Revised: Recent works by

Original: Sato et al.
Revised: Sato et al.

Original: describe the representational power of certain GNN models for tasks such as computing minimum vertex covers (Sato et al., 2019).
Revised: describe the representational power of certain GNN models for tasks such as computing minimum vertex covers (Sato et al., 2019).

Original: Garg et al.
Revised: Garg et al.

Original: also give bounds on the representational power of GNN models, as well as using Rademacher bounds to estimate the generalization ability of
Revised: also give bounds on the representational power of GNN models, as well as using Rademacher bounds to estimate the generalization ability of

Original: GNNs (Garg et al., 2020).
Revised: GNNs (Garg et al., 2020).

Original: Our results concern the task of semi-supervised community detection.
Revised: Our results concern the task of semi-supervised community detection.

Original: In this problem, each vertex belongs to one community, and some subset of the vertices are labeled according to their community membership.
Revised: In this problem, each vertex belongs to one community, and some subset of the vertices are labeled according to their community membership.

Original: The task is to classify the community membership of the remaining vertices.
Revised: The task is to classify the community membership of the remaining vertices.

Original: This task has been one of the most intensively studied problems in the GNN literature, but there have not yet been any provable guarantees on the performance of proposed models.
Revised: This task has been one of the most intensively studied problems in the GNN literature, but there have not yet been any provable guarantees on the performance of proposed models.

Original: We study (spatial-based) graph convolutional models similar to the GCN model proposed in Kipf & Welling (2017).
Revised: We study (spatial-based) graph convolutional models similar to the GCN model proposed in Kipf & Welling (2017).

Original: A single layer of such a model computes weights at each node by aggregating the weights at neighboring nodes and applying an activation function with learned parameters, e.g., a linear map followed by a ReLU.
Revised: A single layer of such a model computes weights at each node by aggregating the weights at neighboring nodes and applying an activation function with learned parameters, e.g., a linear map followed by a ReLU.

Original: Many variations on this theme, including various sophisticated training regimes, have been proposed (Chen et al., 2017; Gao et al., 2018; Li et al., 2018; Zhang et al., 2019b; Chen et al., 2018), but no provable guarantees have been available for the performance of such models on natural data distributions, until the present work.
Revised: Many variations on this theme, including various sophisticated training regimes, have been proposed (Chen et al., 2017; Gao et al., 2018; Li et al., 2018; Zhang et al., 2019b; Chen et al., 2018), but no provable guarantees have been available for the performance of such models on natural data distributions, until the present work.

Original:  M AIN R ESULTS
Revised:  M AIN R ESULTS

Original: One motivation for GNNs as a target for progress in deep learning theory is that there are well-studied graph distributions that plausibly capture some of the structure of real-world data (Drobyshevskiy &
Revised: One motivation for GNNs as a target for progress in deep learning theory is that there are well-studied graph distributions that plausibly capture some of the structure of real-world data (Drobyshevskiy &

Original: Turdakov, 2019).
Revised: Turdakov, 2019).

Original: For example, even fairly simple preferential attachment models plausibly capture some of the essential structure of the web (Kumar et al., 2000).
Revised: For example, even fairly simple preferential attachment models plausibly capture some of the essential structure of the web (Kumar et al., 2000).

Original: Other graph models naturally capture community structures, the simplest of which is the Stochastic Block Model (SBM) (Holland et al., 1983).
Revised: Other graph models naturally capture community structures, the simplest of which is the Stochastic Block Model (SBM) (Holland et al., 1983).

Original: A graph is sampled from a SBM by first partitioning vertices into communities (with fixed or random sizes).
Revised: A graph is sampled from a SBM by first partitioning vertices into communities (with fixed or random sizes).

Original: Two vertices are connected with probability p if they belong to the same community and probability q if they belong to different communities.
Revised: Two vertices are connected with probability p if they belong to the same community and probability q if they belong to different communities.

Original: In this paper, we consider the case of an SBM with two equal-sized communities in which vertices have label 0 and 1 respectively.
Revised: In this paper, we consider the case of an SBM with two equal-sized communities in which vertices have label 0 and 1 respectively.

Original: We denote the label of vertex x by ℓ ( x ) ∈ { 0 , 1 } .
Revised: We denote the label of vertex x by ℓ ( x ) ∈ { 0 , 1 } .

Original: The graphs are parameterized as SBM( n, p, q ) where n is the number of vertices, p is the probability of an intra-community connection, and q is the probability of a cross-community connection.
Revised: The graphs are parameterized as SBM( n, p, q ) where n is the number of vertices, p is the probability of an intra-community connection, and q is the probability of a cross-community connection.

Original: We allow n to vary (but will require it to be sufficiently large), while p and q are of the form p = a log 3 n n and q = b log 3 n n for some fixed constants a > b .
Revised: We allow n to vary (but will require it to be sufficiently large), while p and q are of the form p = a log 3 n n and q = b log 3 n n for some fixed constants a > b .

Original: In the semi-supervised setting, the community labels of some portion of the labels are revealed.
Revised: In the semi-supervised setting, the community labels of some portion of the labels are revealed.

Original: We assume the label of each vertex is revealed independently with probability λ .
Revised: We assume the label of each vertex is revealed independently with probability λ .

Original: The input-layer features at a vertex x is (0 , 0) if its label is not revealed, (1 , 0) if its label is revealed to be 0, and (0 , 1) if its label is revealed to be 1 .
Revised: The input-layer features at a vertex x is (0 , 0) if its label is not revealed, (1 , 0) if its label is revealed to be 0, and (0 , 1) if its label is revealed to be 1 .

Original: Assumption 2.1 (Sparse Stochastic Block Model) .
Revised: Assumption 2.1 (Sparse Stochastic Block Model) .

Original: The probabilities of intra and cross-community connections are p = a log 3 n n and q = b log 3 n n , where a > b are constants.
Revised: The probabilities of intra and cross-community connections are p = a log 3 n n and q = b log 3 n n , where a > b are constants.

Original: We study the problem of recovering the communities from such graphs using GNN models.
Revised: We study the problem of recovering the communities from such graphs using GNN models.

Original: Of course, recovering the communities of an SBM graph has been well-studied and its computational complexity is fully understood in most cases (Abbe & Sandon, 2015; Kawamoto et al., 2019).
Revised: Of course, recovering the communities of an SBM graph has been well-studied and its computational complexity is fully understood in most cases (Abbe & Sandon, 2015; Kawamoto et al., 2019).

Original: SBM models are therefore a natural test-case for understanding the power of GNN models for learning community structure, and experimental studies have been done in this setting (Chen et al., 2020b;
Revised: SBM models are therefore a natural test-case for understanding the power of GNN models for learning community structure, and experimental studies have been done in this setting (Chen et al., 2020b;

Original: Yadav et al., 2019).
Revised: Yadav et al., 2019).

Original: (Abbe et al., 2014) shows a sharp threshold in the task of community recovery: ( √ p − √ q ) (cid:113) n log n > √ 2 .
Revised: (Abbe et al., 2014) shows a sharp threshold in the task of community recovery: ( √ p − √ q ) (cid:113) n log n > √ 2 .

Original: This threshold clearly holds for our case (at sufficiently large values of n ), since p = a log 3 n n , q = b log 3 n n and a > b .
Revised: This threshold clearly holds for our case (at sufficiently large values of n ), since p = a log 3 n n , q = b log 3 n n and a > b .

Original: The contribution here is not to learn the community models.
Revised: The contribution here is not to learn the community models.

Original: Rather it’s showing that (multi-layer)
Revised: Rather it’s showing that (multi-layer)

Original: GCNs solve the classification problem, which is very much not trivial (it is non-convex, and the training loss curve is empirically non-monotonic).
Revised: GCNs solve the classification problem, which is very much not trivial (it is non-convex, and the training loss curve is empirically non-monotonic).

Original: Our GNN models will be trained on a graph or several graphs generated by the SBM ( n, p, q ) model, and seek to understand their accuracy on arbitrary SBM ( n, p, q ) graphs not necessarily in the training set but with the same parameters a, b determining p and q (with n allowed to vary).
Revised: Our GNN models will be trained on a graph or several graphs generated by the SBM ( n, p, q ) model, and seek to understand their accuracy on arbitrary SBM ( n, p, q ) graphs not necessarily in the training set but with the same parameters a, b determining p and q (with n allowed to vary).

Original: In particular, we study spatial-based graph convolutional models along the lines of the Graph
Revised: In particular, we study spatial-based graph convolutional models along the lines of the Graph

Original: Convolutional Networks (GCN) introduced in (Kipf & Welling, 2017).
Revised: Convolutional Networks (GCN) introduced in (Kipf & Welling, 2017).

Original: Each layer of the model computes a feature vector at every vertex of an input graph based on features of nearby vertices in the previous layer.
Revised: Each layer of the model computes a feature vector at every vertex of an input graph based on features of nearby vertices in the previous layer.

Original: A typical layer-wise update rule is of the form
Revised: A typical layer-wise update rule is of the form

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: • ˆ A is a suitably-normalized adjacency matrix of shape n × n where n is the number of vertices.
Revised: • ˆ A is a suitably-normalized adjacency matrix of shape n × n where n is the number of vertices.

Original: Usually ˆ A includes self-loops.
Revised: Usually ˆ A includes self-loops.

Original: • X ( k ) gives the feature vector in the k -th layer at each vertex as a matrix of shape n × m k , where m k is the number of features in layer k .
Revised: • X ( k ) gives the feature vector in the k -th layer at each vertex as a matrix of shape n × m k , where m k is the number of features in layer k .

Original: • ϕ is an activation function, such as the ReLU.
Revised: • ϕ is an activation function, such as the ReLU.

Original: • W ( k ) are the trainable weights in the k -th layer, a matrix of shape m k × m k +1 .
Revised: • W ( k ) are the trainable weights in the k -th layer, a matrix of shape m k × m k +1 .

Original: In our version of this model, we define ˆ A = 1 n 2 ( p + q ) ˜ A, where
Revised: In our version of this model, we define ˆ A = 1 n 2 ( p + q ) ˜ A, where

Original: ˜ A =
Revised: ˜ A =

Original: A + I , A is the adjacency matrix of a given graph, and I is the identity matrix.
Revised: A + I , A is the adjacency matrix of a given graph, and I is the identity matrix.

Original: For the given SBM( n, p, q ), a randomly selected vertex has n 2 ( p + q ) neighbors in expectation, so ˆ A is obtained by normalizing each row of A
Revised: For the given SBM( n, p, q ), a randomly selected vertex has n 2 ( p + q ) neighbors in expectation, so ˆ A is obtained by normalizing each row of A

Original: + I with the average size of a neighborhood.
Revised: + I with the average size of a neighborhood.

Original: Since very deep GCN models seem to provide little empirical benefit (Li et al., 2018), we use a single hidden layer with a softmax output layer.
Revised: Since very deep GCN models seem to provide little empirical benefit (Li et al., 2018), we use a single hidden layer with a softmax output layer.

Original: Furthermore, we introduce a bias term B at the second layer.
Revised: Furthermore, we introduce a bias term B at the second layer.

Original: So the model has the following form:
Revised: So the model has the following form:

Original: [Equation]
Revised: [Equation]

Original: where X is the input feature of the graph and W (0) , W (1) and B are trainable parameters.
Revised: where X is the input feature of the graph and W (0) , W (1) and B are trainable parameters.

Original: Let h denote the number of hidden features, which equals the number of columns of W (0) and the number of rows of W (1) .
Revised: Let h denote the number of hidden features, which equals the number of columns of W (0) and the number of rows of W (1) .

Original: We define the accuracy of the model as the probability of predicting correctly the label of a single vertex in a randomly generated SBM( n, p, q ) graph where the label of each vertex is revealed with probability λ .
Revised: We define the accuracy of the model as the probability of predicting correctly the label of a single vertex in a randomly generated SBM( n, p, q ) graph where the label of each vertex is revealed with probability λ .

Original: We can now state our main result.
Revised: We can now state our main result.

Original: Theorem 2.2.
Revised: Theorem 2.2.

Original: For any ϵ > 0 and δ > 0 , given a GCN model with 1 δ ≤ h ≤ n hidden features and with parameters initialized independently from N (0 , 1) , if training graphs are sampled from
Revised: For any ϵ > 0 and δ > 0 , given a GCN model with 1 δ ≤ h ≤ n hidden features and with parameters initialized independently from N (0 , 1) , if training graphs are sampled from

Original: SBM( n, p, q ) with n ≥ max(Ω( 1 ϵ ) 2 , Ω( 1 δ )) and the label of each vertex revealed with probability λ , and if the model is trained by coordinate descent for k
Revised: SBM( n, p, q ) with n ≥ max(Ω( 1 ϵ ) 2 , Ω( 1 δ )) and the label of each vertex revealed with probability λ , and if the model is trained by coordinate descent for k

Original: = O (log log 1 ϵ ) epochs, then with probability ≥ 1 − δ , the model achieves accuracy ≥ 1 − 4 ϵ .
Revised: = O (log log 1 ϵ ) epochs, then with probability ≥ 1 − δ , the model achieves accuracy ≥ 1 − 4 ϵ .

Original: Remark.
Revised: Remark.

Original: We treat λ as constants, so it is omitted in the big O and Ω notation in the sampling and training complexity.
Revised: We treat λ as constants, so it is omitted in the big O and Ω notation in the sampling and training complexity.

Original: We emphasize that the novelty of this theorem is not in learning two-class SBM models as such; this is a long-solved problem.
Revised: We emphasize that the novelty of this theorem is not in learning two-class SBM models as such; this is a long-solved problem.

Original: Instead, this is the first proof of efficient learning for a GCN on semi-supervised community detection tasks using a natural family of random graph models.
Revised: Instead, this is the first proof of efficient learning for a GCN on semi-supervised community detection tasks using a natural family of random graph models.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this section, we first introduce notations (a table of notations is also shown in the appendix for readers’ convenience) and some interpretations.
Revised: In this section, we first introduce notations (a table of notations is also shown in the appendix for readers’ convenience) and some interpretations.

Original: Then we introduce the structure of the paper.
Revised: Then we introduce the structure of the paper.

Original: Given a vertex y , denote the row of ˜ AX corresponding to y as ( t y 0 , t y 1 ) , so t y 0 and t y 1 give the numbers of neighbors of y (including perhaps y itself) with revealed labels in class 0 and class 1 respectively.
Revised: Given a vertex y , denote the row of ˜ AX corresponding to y as ( t y 0 , t y 1 ) , so t y 0 and t y 1 give the numbers of neighbors of y (including perhaps y itself) with revealed labels in class 0 and class 1 respectively.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Then α i t y 

Revised: Then α i t y

Original: + α ′ i
Revised: + α ′ i

Original: t y 1 , 1 ≤ i ≤ h gives h features of vertex y in the hidden layer.
Revised: t y 1 , 1 ≤ i ≤ h gives h features of vertex y in the hidden layer.

Original: The inner product of the y -th row of ϕ ( ˜ AXW (0) ) and the columns of W (1) gives weighted sums of features of y : (cid:80) hi
Revised: The inner product of the y -th row of ϕ ( ˜ AXW (0) ) and the columns of W (1) gives weighted sums of features of y : (cid:80) hi

Original: =1 β i ϕ
Revised: =1 β i ϕ

Original: ( α i t y 

Revised: ( α i t y

Original: + α ′ i
Revised: + α ′ i

Original: t y 1 ) and (cid:80) hi
Revised: t y 1 ) and (cid:80) hi

Original: =1 β ′ i ϕ
Revised: =1 β ′ i ϕ

Original: ( α i t y 

Revised: ( α i t y

Original: + α ′ i
Revised: + α ′ i

Original: t y 1 ) , where ϕ represents the ReLU function.
Revised: t y 1 ) , where ϕ represents the ReLU function.

Original: Given a vertex x , the row of ˆ Aϕ ( ˆ AXW (0) ) W (1) corresponding to x is denoted by ( f 0 ( x ) , f 1 ( x )) and is of the form
Revised: Given a vertex x , the row of ˆ Aϕ ( ˆ AXW (0) ) W (1) corresponding to x is denoted by ( f 0 ( x ) , f 1 ( x )) and is of the form

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) where 1 [ y ∼ x ] is equal to 1 if y and x are connected, 0 otherwise.
Revised: ) where 1 [ y ∼ x ] is equal to 1 if y and x are connected, 0 otherwise.

Original: Denote f i 0 ( x ) :
Revised: Denote f i 0 ( x ) :

Original: = 4 β i n 2 ( p + q )(cid:88)
Revised: = 4 β i n 2 ( p + q )(cid:88)

Original: [Equation]
Revised: [Equation]

Original: so f 0 ( x ) = (cid:80) hi
Revised: so f 0 ( x ) = (cid:80) hi

Original: =
Revised: =

Original: 1 f i 0 ( x ) and f 1 ( x ) = (cid:80) hi =
Revised: 1 f i 0 ( x ) and f 1 ( x ) = (cid:80) hi =

Original: 1 f i 1 ( x ) .
Revised: 1 f i 1 ( x ) .

Original: Denote g j ( x ) :=
Revised: Denote g j ( x ) :=

Original: f j ( x ) + b j , j = 0 , 1 , where ( g 0 ( x ) , g 1 ( x )) represents the logit of the model corresponding to x .
Revised: f j ( x ) + b j , j = 0 , 1 , where ( g 0 ( x ) , g 1 ( x )) represents the logit of the model corresponding to x .

Original: Denote ∆( x ) :
Revised: Denote ∆( x ) :

Original: = g 0 ( x ) − g 1 ( x ) .
Revised: = g 0 ( x ) − g 1 ( x ) .

Original: In order to make correct predictions, we need ∆( x ) > 0 when ℓ ( x ) = 0 and ∆( x ) < 0 when ℓ ( x ) = 1 .
Revised: In order to make correct predictions, we need ∆( x ) > 0 when ℓ ( x ) = 0 and ∆( x ) < 0 when ℓ ( x ) = 1 .

Original: The bias term B is useful in our analysis because its derivative controls how imbalanced the current loss is between the classes.
Revised: The bias term B is useful in our analysis because its derivative controls how imbalanced the current loss is between the classes.

Original: In training we consider the cross-entropy loss denoted as L , and have
Revised: In training we consider the cross-entropy loss denoted as L , and have

Original: [Equation]
Revised: [Equation]

Original: where Z =
Revised: where Z =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: x )) .
Revised: x )) .

Original: Z can be regarded as a measure of wrong prediction: the numerator is the exponential of the output corresponding to the wrong label and the denominator is a normalizer.
Revised: Z can be regarded as a measure of wrong prediction: the numerator is the exponential of the output corresponding to the wrong label and the denominator is a normalizer.

Original: It is easy to see that Z > 12 if the prediction is wrong; Z < 12 if prediction is correct.
Revised: It is easy to see that Z > 12 if the prediction is wrong; Z < 12 if prediction is correct.

Original: When (cid:12) (cid:12) E [ ∂L∂b 0 ] (cid:12) (cid:12) ≈ 0 , the model’s loss is balanced in the sense of that (cid:12) (cid:12) E
Revised: When (cid:12) (cid:12) E [ ∂L∂b 0 ] (cid:12) (cid:12) ≈ 0 , the model’s loss is balanced in the sense of that (cid:12) (cid:12) E

Original: [ Z | ℓ ( x ) = 0] − E
Revised: [ Z | ℓ ( x ) = 0] − E

Original: [ Z | ℓ ( x ) = 1] (cid:12) (cid:12) ≈ 0 .
Revised: [ Z | ℓ ( x ) = 1] (cid:12) (cid:12) ≈ 0 .

Original: In order to have balanced performance in every epoch, we train the model through coordinate descent instead of conventional gradient descent.
Revised: In order to have balanced performance in every epoch, we train the model through coordinate descent instead of conventional gradient descent.

Original: Specifically, in each epoch we first update b 0 and b 1 until (cid:12)(cid:12) E [ ∂L∂b 0 ] (cid:12)(cid:12) is smaller than some threshold.
Revised: Specifically, in each epoch we first update b 0 and b 1 until (cid:12)(cid:12) E [ ∂L∂b 0 ] (cid:12)(cid:12) is smaller than some threshold.

Original: Then we update the other parameters.
Revised: Then we update the other parameters.

Original: In order to make a learning guarantee of the model, we need a high probability estimation of ∆( x ) .
Revised: In order to make a learning guarantee of the model, we need a high probability estimation of ∆( x ) .

Original: In Section 4, we show that ∆( x ) is concentrated at one of two values, denoted by µ 0 and µ 1 , for ℓ ( x ) = 0 and 1 respectively.
Revised: In Section 4, we show that ∆( x ) is concentrated at one of two values, denoted by µ 0 and µ 1 , for ℓ ( x ) = 0 and 1 respectively.

Original: The proof depends on different parameter regimes of hidden neurons.
Revised: The proof depends on different parameter regimes of hidden neurons.

Original: Furthermore, to avoid the overlap between the concentration range of ∆( x ) , we also show the separation between µ 0 and µ 1 .
Revised: Furthermore, to avoid the overlap between the concentration range of ∆( x ) , we also show the separation between µ 0 and µ 1 .

Original: In Section 5, we analyze the dynamics of hidden neurons throughout training to show that the concentration and separation improve at a controlled rate.
Revised: In Section 5, we analyze the dynamics of hidden neurons throughout training to show that the concentration and separation improve at a controlled rate.

Original: Based on this information, in Section 6 we prove the main theorem.
Revised: Based on this information, in Section 6 we prove the main theorem.

Original: Section 7 shows some experimental results to verify our theory.
Revised: Section 7 shows some experimental results to verify our theory.

Original: The paper ends with future directions in Section 8.
Revised: The paper ends with future directions in Section 8.

Original:  C ONCENTRATION AND S EPARATION OF O UTPUT
Revised:  C ONCENTRATION AND S EPARATION OF O UTPUT

Original: In this section we show that ∆( x ) is concentrated at µ 0 and µ 1 and their separation.
Revised: In this section we show that ∆( x ) is concentrated at µ 0 and µ 1 and their separation.

Original: The difference of the logits is
Revised: The difference of the logits is

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: For brevity, we write ∆( x ) as ∆ and ∆ i ( x ) as ∆ i .
Revised: For brevity, we write ∆( x ) as ∆ and ∆ i ( x ) as ∆ i .

Original: In order to estimate ∆ , we need to estimate each ∆ i , 1 ≤ i ≤ h .
Revised: In order to estimate ∆ , we need to estimate each ∆ i , 1 ≤ i ≤ h .

Original: We denote the high probability estimate of ∆ as µ 0 and µ 1 for ℓ ( x ) = 0 and 1 respectively.
Revised: We denote the high probability estimate of ∆ as µ 0 and µ 1 for ℓ ( x ) = 0 and 1 respectively.

Original: Our fine-grained analysis of the dynamics of coordinate descent on GCNs relies on a classification of neurons into three families based on the sign and scale of the parameters: “good type”, “bad type” and “harmless type”.
Revised: Our fine-grained analysis of the dynamics of coordinate descent on GCNs relies on a classification of neurons into three families based on the sign and scale of the parameters: “good type”, “bad type” and “harmless type”.

Original: The names also indicate whether the neuron has positive contribution to the value of µ 0 − µ 1 .
Revised: The names also indicate whether the neuron has positive contribution to the value of µ 0 − µ 1 .

Original: We show that “good type” neuron makes positive contribution; the contribution of “bad type” neuron is negative but lower bounded; “harmless type” neuron’s contribution is non negative (see Corollary A.4 and the remark following it).
Revised: We show that “good type” neuron makes positive contribution; the contribution of “bad type” neuron is negative but lower bounded; “harmless type” neuron’s contribution is non negative (see Corollary A.4 and the remark following it).

Original: We will specifically describe parameter regime of each type in the following subsections.
Revised: We will specifically describe parameter regime of each type in the following subsections.

Original: We analyze the dynamics of these types throughout coordinate descent in the next section.
Revised: We analyze the dynamics of these types throughout coordinate descent in the next section.

Original: First we give some definitions.
Revised: First we give some definitions.

Original: Definition 1.
Revised: Definition 1.

Original: For 1 ≤ i ≤ h , we call ( α i , α ′ i , β i , β ′ i ) the i -th neuron of the model, where ( α i , α ′ i )
Revised: For 1 ≤ i ≤ h , we call ( α i , α ′ i , β i , β ′ i ) the i -th neuron of the model, where ( α i , α ′ i )

Original: ⊤ is the i -th column of W (0) , ( β i , β ′ i ) is the i -th row of W (1) .
Revised: ⊤ is the i -th column of W (0) , ( β i , β ′ i ) is the i -th row of W (1) .

Original: Definition 2.
Revised: Definition 2.

Original: We say that the i -th neuron is order-aligned if ( α i − α ′ i )( β i − β ′ i ) > 0 , otherwise we say it is order-misaligned .
Revised: We say that the i -th neuron is order-aligned if ( α i − α ′ i )( β i − β ′ i ) > 0 , otherwise we say it is order-misaligned .

Original: 4. C LASSIFICATION OF NEURON PARAMETER REGIMES
Revised: 4. C LASSIFICATION OF NEURON PARAMETER REGIMES

Original: We say the i -th neuron is of “good type” if it satisfies either ( G 1 ) or ( G 2 ) below.
Revised: We say the i -th neuron is of “good type” if it satisfies either ( G 1 ) or ( G 2 ) below.

Original: (There is also the symmetric case obtained by switching α i with α ′ i and β i with β ′ i .
Revised: (There is also the symmetric case obtained by switching α i with α ′ i and β i with β ′ i .

Original: For brevity, we only consider the cases that α i > α ′ i .
Revised: For brevity, we only consider the cases that α i > α ′ i .

Original: This applies to the “bad” and “harmless” types below as well).
Revised: This applies to the “bad” and “harmless” types below as well).

Original: Neurons in this type are order-aligned and
Revised: Neurons in this type are order-aligned and

Original: both α i and α ′ i are positive or the ratio between α i and α ′
Revised: both α i and α ′ i are positive or the ratio between α i and α ′

Original: i is large enough.
Revised: i is large enough.

Original: [Equation]
Revised: [Equation]

Original: We say the i -th neuron is of “bad type” if it satisfies either ( B 1 ), ( B 2 ) or ( B 3 ).
Revised: We say the i -th neuron is of “bad type” if it satisfies either ( B 1 ), ( B 2 ) or ( B 3 ).

Original: Neurons in this type are order-misaligned and α i , α ′ i are either both positive or have the opposite signs.
Revised: Neurons in this type are order-misaligned and α i , α ′ i are either both positive or have the opposite signs.

Original: [Equation]
Revised: [Equation]

Original: B
Revised: B

Original: [Equation]
Revised: [Equation]

Original: We say that the i -th neuron is of “harmless type” if it satisfies either ( H 1 ) or ( H 2 ): α i > 0 >
Revised: We say that the i -th neuron is of “harmless type” if it satisfies either ( H 1 ) or ( H 2 ): α i > 0 >

Original: α ′ i , (cid:12)(cid:12)(cid:12)(cid:12)
Revised: α ′ i , (cid:12)(cid:12)(cid:12)(cid:12)

Original: α i α ′ i (cid:12)(cid:12)(cid:12)(cid:12) ∈ ( q p (1 + log − 13 n ) , 1] and β i > β ′
Revised: α i α ′ i (cid:12)(cid:12)(cid:12)(cid:12) ∈ ( q p (1 + log − 13 n ) , 1] and β i > β ′

Original: i ( H 1 ) α i ≤ 0 and α ′ i
Revised: i ( H 1 ) α i ≤ 0 and α ′ i

Original: ≤( H 2 )
Revised: ≤( H 2 )

Original: 4. C ONCENTRATION AND S EPARATION
Revised: 4. C ONCENTRATION AND S EPARATION

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: If the i -th neuron is of “good type” satisfying ( G 1 ) or of “bad type” satisfying ( B 1 ), then for ℓ ( x ) = 0 :
Revised: If the i -th neuron is of “good type” satisfying ( G 1 ) or of “bad type” satisfying ( B 1 ), then for ℓ ( x ) = 0 :

Original: [Equation]
Revised: [Equation]

Original: for ℓ ( x ) = 1 :
Revised: for ℓ ( x ) = 1 :

Original: [Equation]
Revised: [Equation]

Original: Similar concentration hold for neurons satisfying ( G 2 ), ( B 2 ) and ( B 3 ), and for neurons of “harmless type.”
Revised: Similar concentration hold for neurons satisfying ( G 2 ), ( B 2 ) and ( B 3 ), and for neurons of “harmless type.”

Original: We apply the method of bounded differences to show the concentration.
Revised: We apply the method of bounded differences to show the concentration.

Original: The details are shown in the appendix.
Revised: The details are shown in the appendix.

Original: Given the concentration of ∆ i for each type of neurons, we estimate the concentration of the output ∆ =
Revised: Given the concentration of ∆ i for each type of neurons, we estimate the concentration of the output ∆ =

Original: (cid:80) hi
Revised: (cid:80) hi

Original: =1 ∆ i + b 0 − b 1 .
Revised: =1 ∆ i + b 0 − b 1 .

Original: For the i -th neuron, we denote the high-probability estimate of ∆ i given in the statement of Theorem 4.1 as m i 0 when ℓ ( x ) = 

Revised: For the i -th neuron, we denote the high-probability estimate of ∆ i given in the statement of Theorem 4.1 as m i 0 when ℓ ( x ) =

Original: and m i 1 when ℓ ( x ) = 1 .
Revised: and m i 1 when ℓ ( x ) = 1 .

Original: By union bound, we have the following corollary.
Revised: By union bound, we have the following corollary.

Original: Corollary 4.2.
Revised: Corollary 4.2.

Original: Given a vertex x ∈ G with label unrevealed, we have
Revised: Given a vertex x ∈ G with label unrevealed, we have

Original: [Equation]
Revised: [Equation]

Original: where µ
Revised: where µ

Original: [Equation]
Revised: [Equation]

Original: For any ϵ > 0 , we require the probability of concentration in (3) to be at least 1 − ˜ ϵ , where ˜ ϵ
Revised: For any ϵ > 0 , we require the probability of concentration in (3) to be at least 1 − ˜ ϵ , where ˜ ϵ

Original: = o ( ϵ ) .
Revised: = o ( ϵ ) .

Original: If we choose ˜ ϵ
Revised: If we choose ˜ ϵ

Original: = ϵ 2 , then we set 1 − O ( 1 n ) ≥ 1 − ϵ 2 , i.e. n ≥ Ω( 1 ϵ ) 2 .
Revised: = ϵ 2 , then we set 1 − O ( 1 n ) ≥ 1 − ϵ 2 , i.e. n ≥ Ω( 1 ϵ ) 2 .

Original: Our following analysis will be based on this condition.
Revised: Our following analysis will be based on this condition.

Original: From Theorem 4.1, we have the following result about the value of m i 0 − m i 1 .
Revised: From Theorem 4.1, we have the following result about the value of m i 0 − m i 1 .

Original: Corollary 4.3.
Revised: Corollary 4.3.

Original: • If the i -th neuron is of “good type” and satisfies ( G 1 ), then
Revised: • If the i -th neuron is of “good type” and satisfies ( G 1 ), then

Original: [Equation]
Revised: [Equation]

Original: • If the i -th neuron is of “bad type” and satisfies ( B 1 ), then
Revised: • If the i -th neuron is of “bad type” and satisfies ( B 1 ), then

Original: [Equation]
Revised: [Equation]

Original: • If the i -th neuron is of “harmless type” and satisfies ( H 1 ), then
Revised: • If the i -th neuron is of “harmless type” and satisfies ( H 1 ), then

Original: [Equation]
Revised: [Equation]

Original: Similar results for neurons satisfying ( G 2 ),( B 2 ),( B 3 ) and ( H 1 ) are stated in the appendix, along with the proof.
Revised: Similar results for neurons satisfying ( G 2 ),( B 2 ),( B 3 ) and ( H 1 ) are stated in the appendix, along with the proof.

Original: Remark.
Revised: Remark.

Original: • As we can see from Corollary 4.3, the value of m i 0 − m i 1 is positive for “good type” neurons, non-negative for “harmless type” neurons and may be negative (but lower bounded) for “bad type” neurons.
Revised: • As we can see from Corollary 4.3, the value of m i 0 − m i 1 is positive for “good type” neurons, non-negative for “harmless type” neurons and may be negative (but lower bounded) for “bad type” neurons.

Original: Since positive values of m i 0 − m i 1 decrease the loss of the model, this explains the names for the types of neurons.
Revised: Since positive values of m i 0 − m i 1 decrease the loss of the model, this explains the names for the types of neurons.

Original: • m i 0 − m i 1 is proportional to | α
Revised: • m i 0 − m i 1 is proportional to | α

Original: i − α
Revised: i − α

Original: i
Revised: i

Original: ||
Revised: ||

Original: β
Revised: β

Original: i − β ′ i | .
Revised: i − β ′ i | .

Original: In the next section, we analyze the dynamics of the parameters α i , α ′ i , β i , β ′ i .
Revised: In the next section, we analyze the dynamics of the parameters α i , α ′ i , β i , β ′ i .

Original: Using our understanding of these dynamics, in Theorem 6.2 we present a refined result about the separation of output which only depends on the initialization of parameters.
Revised: Using our understanding of these dynamics, in Theorem 6.2 we present a refined result about the separation of output which only depends on the initialization of parameters.

Original: • Let c := µ 0 − µ 

Revised: • Let c := µ 0 − µ

Original: = (cid:80) hi
Revised: = (cid:80) hi

Original: =1 ( m i 0 − m i 1 ) .
Revised: =1 ( m i 0 − m i 1 ) .

Original: By the two corollaries above, we have δ =
Revised: By the two corollaries above, we have δ =

Original: o ( | c | ) .
Revised: o ( | c | ) .

Original: The balanced loss guaranteed by the
Revised: The balanced loss guaranteed by the

Original: bias term and the coordinate descent scheme ensure that µ 0 = Ω( c ) and µ 1 = Ω( c ) .
Revised: bias term and the coordinate descent scheme ensure that µ 0 = Ω( c ) and µ 1 = Ω( c ) .

Original: It then follows that if the loss is sufficiently small, both µ 0 and µ 1 have correct sign, i.e. µ 0 > 0 > µ 1 .
Revised: It then follows that if the loss is sufficiently small, both µ 0 and µ 1 have correct sign, i.e. µ 0 > 0 > µ 1 .

Original: (Otherwise, due to concentration of the output, the model makes wrong prediction and the loss is large).
Revised: (Otherwise, due to concentration of the output, the model makes wrong prediction and the loss is large).

Original: So we will eventually have δ =
Revised: So we will eventually have δ =

Original: o ( µ 0 ) and δ =
Revised: o ( µ 0 ) and δ =

Original: o ( | µ 1 | ) .
Revised: o ( | µ 1 | ) .

Original:  D YNAMICS OF P ARAMETERS
Revised:  D YNAMICS OF P ARAMETERS

Original: In this section, we describe the dynamics of each type of neurons through coordinate descent, which can be visualized in the following figure in which the arrows indicate movement between types that can happen with non-negligible probability.
Revised: In this section, we describe the dynamics of each type of neurons through coordinate descent, which can be visualized in the following figure in which the arrows indicate movement between types that can happen with non-negligible probability.

Original: [Figure]
Revised: [Figure]

Original: There are two noteworthy points from this figure.
Revised: There are two noteworthy points from this figure.

Original: First, “good type” parameters are preserved under coordinate descent.
Revised: First, “good type” parameters are preserved under coordinate descent.

Original: Second, there are no arrows coming into “bad type” except from itself.
Revised: Second, there are no arrows coming into “bad type” except from itself.

Original: These dynamics are proved by estimating the gradient with respect to the loss function for each type of neuron.
Revised: These dynamics are proved by estimating the gradient with respect to the loss function for each type of neuron.

Original: Because of the non-linearity of the activation, we rely heavily on the concentration result proved above to get tight estimates.
Revised: Because of the non-linearity of the activation, we rely heavily on the concentration result proved above to get tight estimates.

Original: Without these concentration results, even estimating the sign of the gradient seems difficult.
Revised: Without these concentration results, even estimating the sign of the gradient seems difficult.

Original: The proof and experiments about the dynamics of hidden neurons are deferred to the appendix.
Revised: The proof and experiments about the dynamics of hidden neurons are deferred to the appendix.

Original:  L EARNING G UARANTEE
Revised:  L EARNING G UARANTEE

Original: In this section, we prove our main result which states that with high probability a trained GCN can detect communities in SBM with any desired accuracy.
Revised: In this section, we prove our main result which states that with high probability a trained GCN can detect communities in SBM with any desired accuracy.

Original: The proof is based on the following theorem which shows that if µ 0 and µ 1 are separated enough, then the model achieves high accuracy.
Revised: The proof is based on the following theorem which shows that if µ 0 and µ 1 are separated enough, then the model achieves high accuracy.

Original: Theorem 6.1.
Revised: Theorem 6.1.

Original: ∀ ϵ > 0 , provided that the difference between µ 0 and µ 1 is large enough: σ ( − µ 0 − µ 1) <
Revised: ∀ ϵ > 0 , provided that the difference between µ 0 and µ 1 is large enough: σ ( − µ 0 − µ 1) <

Original: ϵ 2 , if (cid:12)(cid:12) E
Revised: ϵ 2 , if (cid:12)(cid:12) E

Original: [ ∂L∂b 0 ] (cid:12)(cid:12) <
Revised: [ ∂L∂b 0 ] (cid:12)(cid:12) <

Original: ϵ 4 , then P [∆ < 0 | ℓ ( x ) = 0] < 4 ϵ, P [∆ > 0 | ℓ ( x ) = 1] < 4 ϵ , where σ ( x ) :
Revised: ϵ 4 , then P [∆ < 0 | ℓ ( x ) = 0] < 4 ϵ, P [∆ > 0 | ℓ ( x ) = 1] < 4 ϵ , where σ ( x ) :

Original: = 1 / (1 + exp ( − x )) represents the sigmoid function.
Revised: = 1 / (1 + exp ( − x )) represents the sigmoid function.

Original: Next we show that the model can achieve such separation between µ 0 and µ 1 through coordinate descent.
Revised: Next we show that the model can achieve such separation between µ 0 and µ 1 through coordinate descent.

Original: In order to make constant update of parameters at every epoch, we set an adaptive learning rate η k = 1 E [ Z ( k ) ] where Z ( k ) is the value of Z at the k -th epoch.
Revised: In order to make constant update of parameters at every epoch, we set an adaptive learning rate η k = 1 E [ Z ( k ) ] where Z ( k ) is the value of Z at the k -th epoch.

Original: We first refine Corollary 4.3 about the separation of output for each type of neuron ( m i 0 − m i 1 ) using the dynamics of parameters.
Revised: We first refine Corollary 4.3 about the separation of output for each type of neuron ( m i 0 − m i 1 ) using the dynamics of parameters.

Original: Theorem 6.2 (separation of output) .
Revised: Theorem 6.2 (separation of output) .

Original: Let m i 

Revised: Let m i

Original: and m i 1 be defined as in Section 4, train the model for k epochs by the defined coordinate descent with adaptive learning rate η k =
Revised: and m i 1 be defined as in Section 4, train the model for k epochs by the defined coordinate descent with adaptive learning rate η k =

Original: [Equation]
Revised: [Equation]

Original: • if the i -th neuron is of “good type”, then
Revised: • if the i -th neuron is of “good type”, then

Original: [Equation]
Revised: [Equation]

Original: • if the i -th neuron is of “bad type”, then
Revised: • if the i -th neuron is of “bad type”, then

Original: [Equation]
Revised: [Equation]

Original: • if the i -th neuron is of “harmless type”, then
Revised: • if the i -th neuron is of “harmless type”, then

Original: [Equation]
Revised: [Equation]

Original: where A (0)
Revised: where A (0)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Next we present a result about initialization, which shows that with high probability, there are enough “good type” neurons and parameters have appropriate scale.
Revised: Next we present a result about initialization, which shows that with high probability, there are enough “good type” neurons and parameters have appropriate scale.

Original: Lemma 6.3.
Revised: Lemma 6.3.

Original: Suppose all parameters in W (0) and W (1) are initialized independently following standard normal distribution.
Revised: Suppose all parameters in W (0) and W (1) are initialized independently following standard normal distribution.

Original: Then the number h g of neurons initialized as “good type” satisfies
Revised: Then the number h g of neurons initialized as “good type” satisfies

Original: P [
Revised: P [

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1 − exp ( −
Revised: 1 − exp ( −

Original: [Equation]
Revised: [Equation]

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: the i
Revised: the i

Original: [Equation]
Revised: [Equation]

Original: neuron initialized as “good
Revised: neuron initialized as “good

Original: [Equation]
Revised: [Equation]

Original: Now we can prove the final result.
Revised: Now we can prove the final result.

Original: Proof of Theorem 2.2.
Revised: Proof of Theorem 2.2.

Original: First we show that if the loss E [ Z ] is small enough, the model achieves desired accuracy.
Revised: First we show that if the loss E [ Z ] is small enough, the model achieves desired accuracy.

Original: Indeed, if E [ Z ] < 2 ϵ , since
Revised: Indeed, if E [ Z ] < 2 ϵ , since

Original: E [ Z ] =
Revised: E [ Z ] =

Original: E [ Z | pred is wrong ]
Revised: E [ Z | pred is wrong ]

Original: P [ pred is wrong ]+ E
Revised: P [ pred is wrong ]+ E

Original: [ Z | pred is correct ] P [ pred is correct ] ≥ 1 2 P [ pred is wrong ] , we have P [ pred is wrong ] ≤ 4 ϵ , i.e., P [ pred is correct ] > 1 − 4 ϵ .
Revised: [ Z | pred is correct ] P [ pred is correct ] ≥ 1 2 P [ pred is wrong ] , we have P [ pred is wrong ] ≤ 4 ϵ , i.e., P [ pred is correct ] > 1 − 4 ϵ .

Original: Otherwise, E [ Z ] ≥ 2 ϵ , since E [ Z ] = 12 ( E [ Z | ℓ ( z ) = 0]
Revised: Otherwise, E [ Z ] ≥ 2 ϵ , since E [ Z ] = 12 ( E [ Z | ℓ ( z ) = 0]

Original: + E [ Z | ℓ ( z ) = 1]) , we have E [ Z | ℓ ( z ) =
Revised: + E [ Z | ℓ ( z ) = 1]) , we have E [ Z | ℓ ( z ) =

Original: 0]+
Revised: 0]+

Original: E [ Z | ℓ ( z ) = 1] ≥ 4 ϵ .
Revised: E [ Z | ℓ ( z ) = 1] ≥ 4 ϵ .

Original: On the other hand, (cid:12)(cid:12) E [ ∂L∂b 0 ] (cid:12)(cid:12) <
Revised: On the other hand, (cid:12)(cid:12) E [ ∂L∂b 0 ] (cid:12)(cid:12) <

Original: ϵ implies that (cid:12)(cid:12) E
Revised: ϵ implies that (cid:12)(cid:12) E

Original: [ Z | ℓ ( z ) = 0] − E
Revised: [ Z | ℓ ( z ) = 0] − E

Original: [ Z | ℓ ( z ) = 1] (cid:12)(cid:12) <
Revised: [ Z | ℓ ( z ) = 1] (cid:12)(cid:12) <

Original: 2 ϵ.
Revised: 2 ϵ.

Original: By Theorem 6.2, µ 0 − µ 1 = h (cid:88) i =
Revised: By Theorem 6.2, µ 0 − µ 1 = h (cid:88) i =

Original: ( m i
Revised: ( m i

Original: [Equation]
Revised: [Equation]

Original: m i
Revised: m i

Original: [Equation]
Revised: [Equation]

Original: = (cid:88)
Revised: = (cid:88)

Original: i ∈ “good” ( m
Revised: i ∈ “good” ( m

Original: [Equation]
Revised: [Equation]

Original: 0 − m
Revised: 0 − m

Original: [Equation]
Revised: [Equation]

Original: 1 ) +
Revised: 1 ) +

Original: [Equation]
Revised: [Equation]

Original: i ∈ “bad” ( m
Revised: i ∈ “bad” ( m

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: 1 ) + (cid:88) i ∈ “harmless” ( m i 

Revised: 1 ) + (cid:88) i ∈ “harmless” ( m i

Original: − m i 1 ) ≥ λ (cid:18) p − q p + q (cid:19) 2 (cid:18) 1 + √ 2 λ (cid:18) p
Revised: − m i 1 ) ≥ λ (cid:18) p − q p + q (cid:19) 2 (cid:18) 1 + √ 2 λ (cid:18) p

Original: [Equation]
Revised: [Equation]

Original: q p + q (cid:19) 2 (cid:19) 2 k (cid:88) i ∈
Revised: q p + q (cid:19) 2 (cid:19) 2 k (cid:88) i ∈

Original: “good”
Revised: “good”

Original: A (0) i B (0)
Revised: A (0) i B (0)

Original: i − k λp ( p − q ) ( p + q )(cid:88) i ∈ “bad” (cid:0)(cid:0) A (0) i (cid:1) 

Revised: i − k λp ( p − q ) ( p + q )(cid:88) i ∈ “bad” (cid:0)(cid:0) A (0) i (cid:1)

Original: + (cid:0) B (0) i (cid:1) 2 (cid:1) .
Revised: + (cid:0) B (0) i (cid:1) 2 (cid:1) .

Original: By Lemma 6.3, with probability ≥ 1 − O ( 1 h ) , (cid:88) i ∈ “good”
Revised: By Lemma 6.3, with probability ≥ 1 − O ( 1 h ) , (cid:88) i ∈ “good”

Original: A (0) i B (0)
Revised: A (0) i B (0)

Original: i ≥ h 80 , (cid:88) i ∈
Revised: i ≥ h 80 , (cid:88) i ∈

Original: “bad” (cid:0)(cid:0) A (0) i (cid:1) 

Revised: “bad” (cid:0)(cid:0) A (0) i (cid:1)

Original: + (cid:0) B (0) i (cid:1) 2 (cid:1) ≤ 5 h.
Revised: + (cid:0) B (0) i (cid:1) 2 (cid:1) ≤ 5 h.

Original: Since h ≥ 1 δ , then with probability ≥ 1 − δ , µ 0 − µ 1 ≥ h (cid:18) λ (cid:18) p − q p + q (cid:19) 2 (cid:18) 1 + √ 2 λ (cid:18) p − q p + q (cid:19)
Revised: Since h ≥ 1 δ , then with probability ≥ 1 − δ , µ 0 − µ 1 ≥ h (cid:18) λ (cid:18) p − q p + q (cid:19) 2 (cid:18) 1 + √ 2 λ (cid:18) p − q p + q (cid:19)

Original: [Equation]
Revised: [Equation]

Original: where C 1 , C 2 and C 3 are constants determined by p, q and λ .
Revised: where C 1 , C 2 and C 3 are constants determined by p, q and λ .

Original: By Theorem 6.1, if (4) ≥ 2 log 2 ϵ (then σ ( − µ 0 − µ 1) ≤ ϵ 2 ), then the model achieves accuracy ≥ 1 − 4 ϵ .
Revised: By Theorem 6.1, if (4) ≥ 2 log 2 ϵ (then σ ( − µ 0 − µ 1) ≤ ϵ 2 ), then the model achieves accuracy ≥ 1 − 4 ϵ .

Original: It’s sufficient to have
Revised: It’s sufficient to have

Original: [Equation]
Revised: [Equation]

Original: i.e. k =
Revised: i.e. k =

Original: O (log log 1 ϵ ) .
Revised: O (log log 1 ϵ ) .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We show some experiments verifying Theorem 2.2.
Revised: We show some experiments verifying Theorem 2.2.

Original: In particular, our experiments demonstrate that accuracy increases with n , the probability of high-accuracy models increases with h , and coordinate descent is able to recovery high-accuracy models in the sparse regime of Assumption 2.1.
Revised: In particular, our experiments demonstrate that accuracy increases with n , the probability of high-accuracy models increases with h , and coordinate descent is able to recovery high-accuracy models in the sparse regime of Assumption 2.1.

Original: Additional plots demonstrating the dynamics of hidden neurons with their ratios and differences can be seen in the appendix.
Revised: Additional plots demonstrating the dynamics of hidden neurons with their ratios and differences can be seen in the appendix.

Original: Experiment 1 In this experiment, we plot the an estimate of the accuracy versus epoch for varying n .
Revised: Experiment 1 In this experiment, we plot the an estimate of the accuracy versus epoch for varying n .

Original: The parameters p, q of SBM follow
Revised: The parameters p, q of SBM follow

Original: Assumption 2.1, where we choose a = 1 . 0 and b = 0 .
Revised: Assumption 2.1, where we choose a = 1 . 0 and b = 0 .

Original: We set h = 20 , λ = 0 .
Revised: We set h = 20 , λ = 0 .

Original: 3 and run 40 independent experiments for n = 250 ,and 1000 respectively.
Revised: 3 and run 40 independent experiments for n = 250 ,and 1000 respectively.

Original: In each experiment we train the model for 100 epochs.
Revised: In each experiment we train the model for 100 epochs.

Original: The training set has 40 randomly generated graphs from
Revised: The training set has 40 randomly generated graphs from

Original: SBM ( n, p, q ) .
Revised: SBM ( n, p, q ) .

Original: We validate the performance by the percentage of correct predictions on 200 random vertices, each from a randomly generated graph.
Revised: We validate the performance by the percentage of correct predictions on 200 random vertices, each from a randomly generated graph.

Original: The result is shown Figure 2.
Revised: The result is shown Figure 2.

Original: The shaded region for each n is obtained from the max, min and mean percentage of the 40 experiments.
Revised: The shaded region for each n is obtained from the max, min and mean percentage of the 40 experiments.

Original: The result verifies Theorem 2.2 which shows that the accuracy of the model increases with n .
Revised: The result verifies Theorem 2.2 which shows that the accuracy of the model increases with n .

Original: Experiment 2 In this experiment, we show the effect of the number of hidden neurons h .
Revised: Experiment 2 In this experiment, we show the effect of the number of hidden neurons h .

Original: The parameters of SBM are the same as Experiment 1.
Revised: The parameters of SBM are the same as Experiment 1.

Original: We set h = 2 , 5 , 20 .
Revised: We set h = 2 , 5 , 20 .

Original: For each pair of ( n, h ) we run 40 independent experiments and show the distribution of validation in Figure 3.
Revised: For each pair of ( n, h ) we run 40 independent experiments and show the distribution of validation in Figure 3.

Original: From the top row to the bottom, n increases fromto 1000.
Revised: From the top row to the bottom, n increases fromto 1000.

Original: From the left column to the right, h increases from 2 to 20.
Revised: From the left column to the right, h increases from 2 to 20.

Original: In each plot, the x-axis represents the accuracy, while y-axis represents the count of experiments.
Revised: In each plot, the x-axis represents the accuracy, while y-axis represents the count of experiments.

Original: According to Theorem 2.2, the probability of achieving high accuracy is 1 − O (1 /h ) and the accuracy increases with n .
Revised: According to Theorem 2.2, the probability of achieving high accuracy is 1 − O (1 /h ) and the accuracy increases with n .

Original: We can see that in each row of Figure 3, as h increases, we have lager probability to achieve high accuracy; in each column, as n increases, the model achieves higher accuracy.
Revised: We can see that in each row of Figure 3, as h increases, we have lager probability to achieve high accuracy; in each column, as n increases, the model achieves higher accuracy.

Original: The results verify our theory in the paper.
Revised: The results verify our theory in the paper.

Original:  F UTURE D IRECTIONS
Revised:  F UTURE D IRECTIONS

Original: Graph neural networks offer a promising setting for progress on the more general theory of deep learning, because random graph models more plausibly capture the structure of real-world data compared to, e.g., the Gaussian inputs often used to prove deep learning guarantees for traditional feed-forward neural networks.
Revised: Graph neural networks offer a promising setting for progress on the more general theory of deep learning, because random graph models more plausibly capture the structure of real-world data compared to, e.g., the Gaussian inputs often used to prove deep learning guarantees for traditional feed-forward neural networks.

Original: This paper has initiated the project of proving training guarantees for semi-supervised learning using GCNs on SBM models, but much more work remains to be done.
Revised: This paper has initiated the project of proving training guarantees for semi-supervised learning using GCNs on SBM models, but much more work remains to be done.

Original: Arguably the sparsest SBM models (expected constant degree) are the most compelling from the perspective of modeling real-world communities, so it would be interesting to extend these results to that setting.
Revised: Arguably the sparsest SBM models (expected constant degree) are the most compelling from the perspective of modeling real-world communities, so it would be interesting to extend these results to that setting.

Original: Models with more than two blocks, or overlapping communities (Petti & Vempala, 2018) would be even closer to real-world structure.
Revised: Models with more than two blocks, or overlapping communities (Petti & Vempala, 2018) would be even closer to real-world structure.

Original: We hope this initial step spurs further interest in provable guarantees for training neural networks using plausible models of real-world data as the input distribution.
Revised: We hope this initial step spurs further interest in provable guarantees for training neural networks using plausible models of real-world data as the input distribution.

Original:  A CKNOWLEDGEMENT
Revised:  A CKNOWLEDGEMENT

Original: This research was supported in part by NSF Grant 1849796.
Revised: This research was supported in part by NSF Grant 1849796.

Original: The author would like to thank Prof.
Revised: The author would like to thank Prof.

Original: John Wilmes for guiding the research, and also thank Prof. Tyler Maunu and Prof. Pengyu Hong for making the results more generalized and designing the experiments.
Revised: John Wilmes for guiding the research, and also thank Prof. Tyler Maunu and Prof. Pengyu Hong for making the results more generalized and designing the experiments.

Original: A C ONCENTRATION AND S EPARATION OF O UTPUT
Revised: A C ONCENTRATION AND S EPARATION OF O UTPUT

Original: Let N ( x, 0) , N ( x, 1) denote the neighborhood of vertex x with label 0 and 1 respectively, i.e.
Revised: Let N ( x, 0) , N ( x, 1) denote the neighborhood of vertex x with label 0 and 1 respectively, i.e.

Original: N ( x, 0) = { y ∈ G, y ∼ x, ℓ ( y ) = 0 } , N ( x, 1) =
Revised: N ( x, 0) = { y ∈ G, y ∼ x, ℓ ( y ) = 0 } , N ( x, 1) =

Original: { y ∈ G, y ∼ x, ℓ ( y )
Revised: { y ∈ G, y ∼ x, ℓ ( y )

Original: = 1 } .
Revised: = 1 } .

Original: By the definition of SBM, both | N ( x, 0) | and | N ( x, 1) | are binomial random variables.
Revised: By the definition of SBM, both | N ( x, 0) | and | N ( x, 1) | are binomial random variables.

Original: For ℓ ( x ) = 0 , | N ( x, 0) | ∼
Revised: For ℓ ( x ) = 0 , | N ( x, 0) | ∼

Original: B ( n 2 , p ) , | N ( x, 1) | ∼ B ( n 2 , q ) and for ℓ ( x ) = 1 , | N ( x, 0) | ∼ B ( n 2 , q ) , | N ( x, 1) | ∼ B ( n 2 , p ) .
Revised: B ( n 2 , p ) , | N ( x, 1) | ∼ B ( n 2 , q ) and for ℓ ( x ) = 1 , | N ( x, 0) | ∼ B ( n 2 , q ) , | N ( x, 1) | ∼ B ( n 2 , p ) .

Original: Moreover, t y 0 and t y 1 are also binomial random variables, for ℓ ( y )
Revised: Moreover, t y 0 and t y 1 are also binomial random variables, for ℓ ( y )

Original: = 0 , t y 0 ∼ B ( nλ 2 , p ) , t y 1 ∼ B ( nλ 2 , q ) , similarly for ℓ ( y ) = 1 .
Revised: = 0 , t y 0 ∼ B ( nλ 2 , p ) , t y 1 ∼ B ( nλ 2 , q ) , similarly for ℓ ( y ) = 1 .

Original: Our following analysis is based on the condition that | N ( x, 0) | , | N ( x, 1) | , t xand t x 1 are in their high probability range for all x ∈ G .
Revised: Our following analysis is based on the condition that | N ( x, 0) | , | N ( x, 1) | , t xand t x 1 are in their high probability range for all x ∈ G .

Original: Specifically we require the condition that for all ℓ ( x ) = 0 , (similar conditions for ℓ ( x ) = 1 are omitted):
Revised: Specifically we require the condition that for all ℓ ( x ) = 0 , (similar conditions for ℓ ( x ) = 1 are omitted):

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: By tail bound of binomial random variables and union bound, we have
Revised: By tail bound of binomial random variables and union bound, we have

Original: [Equation]
Revised: [Equation]

Original: Under this condition, we show the concentration of ∆ i for each type.
Revised: Under this condition, we show the concentration of ∆ i for each type.

Original: A. “G OOD T YPE ” NEURONS
Revised: A. “G OOD T YPE ” NEURONS

Original: For convenience, according to the activation pattern of ϕ ( α i t y 

Revised: For convenience, according to the activation pattern of ϕ ( α i t y

Original: + α ′ i
Revised: + α ′ i

Original: t y 1 ) , we further divide ( G 2 ) into subcases ( G 2 , 1 ), ( G 2 , 2 ) and ( G 2 , 3 ) by to the ratio of (cid:12)(cid:12)
Revised: t y 1 ) , we further divide ( G 2 ) into subcases ( G 2 , 1 ), ( G 2 , 2 ) and ( G 2 , 3 ) by to the ratio of (cid:12)(cid:12)

Original: α i α ′ i (cid:12)(cid:12) .
Revised: α i α ′ i (cid:12)(cid:12) .

Original: For example, in ( G 1 ) and ( G 2 , 1 ), ϕ ( α i t y 

Revised: For example, in ( G 1 ) and ( G 2 , 1 ), ϕ ( α i t y

Original: + α ′ i
Revised: + α ′ i

Original: t y 1 ) is active for both ℓ ( y )
Revised: t y 1 ) is active for both ℓ ( y )

Original: = 0 and ℓ ( y )
Revised: = 0 and ℓ ( y )

Original: = 1 ; in ( G 2 , 1 ), it is only active for ℓ ( y )
Revised: = 1 ; in ( G 2 , 1 ), it is only active for ℓ ( y )

Original: = 0 .
Revised: = 0 .

Original: [Equation]
Revised: [Equation]

Original: We have the following estimation of ∆ i in “good type”.
Revised: We have the following estimation of ∆ i in “good type”.

Original: Theorem A.1 (concentration of output from “good type” neurons) .
Revised: Theorem A.1 (concentration of output from “good type” neurons) .

Original: If the i -th neuron is of “good type”, then
Revised: If the i -th neuron is of “good type”, then

Original: • in both ( G 1 ) and ( G 2 , 1 ):
Revised: • in both ( G 1 ) and ( G 2 , 1 ):

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: P [ (cid:12)(cid:12)
Revised: P [ (cid:12)(cid:12)

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: • in ( G 2 , 2 ): P [ (cid:12)(cid:12) ∆ i − λ
Revised: • in ( G 2 , 2 ): P [ (cid:12)(cid:12) ∆ i − λ

Original: [Table]
Revised: [Table]

Original: β i
Revised: β i

Original: [Table]
Revised: [Table]

Original: P [ (cid:12)(cid:12) ∆ i
Revised: P [ (cid:12)(cid:12) ∆ i

Original: [Table]
Revised: [Table]

Original: λ
Revised: λ

Original: [Table]
Revised: [Table]

Original: • in ( G 2 , 3 ): P [ (cid:12)(cid:12) ∆ i − λ
Revised: • in ( G 2 , 3 ): P [ (cid:12)(cid:12) ∆ i − λ

Original: ( β i − β ′ i ) p ( p + q ) 2 (
Revised: ( β i − β ′ i ) p ( p + q ) 2 (

Original: pα i + qα ′ i ) (cid:12)(cid:12) ≤ ( α i − α ′ i )
Revised: pα i + qα ′ i ) (cid:12)(cid:12) ≤ ( α i − α ′ i )

Original: ( β i − β ′ i )
Revised: ( β i − β ′ i )

Original: O
Revised: O

Original: [Table]
Revised: [Table]

Original: log −
Revised: log −

Original: [Table]
Revised: [Table]

Original: ) | ℓ
Revised: ) | ℓ

Original: [Table]
Revised: [Table]

Original: x ) = 0] ≥
Revised: x ) = 0] ≥

Original: [Table]
Revised: [Table]

Original: O
Revised: O

Original: [Table]
Revised: [Table]

Original: 1 n
Revised: 1 n

Original: [Table]
Revised: [Table]

Original: ) P [ (cid:12)(cid:12) ∆ i − λ ( β i − β ′ i ) q ( p + q ) 2 ( pα i + qα ′ i ) (cid:12)(cid:12)
Revised: ) P [ (cid:12)(cid:12) ∆ i − λ ( β i − β ′ i ) q ( p + q ) 2 ( pα i + qα ′ i ) (cid:12)(cid:12)

Original: ≤ ( α i − α ′ i )
Revised: ≤ ( α i − α ′ i )

Original: ( β i − β ′ i )
Revised: ( β i − β ′ i )

Original: O (log − 12 n ) | ℓ ( x ) = 1] ≥ 1 − O ( 1 n 2 ) .
Revised: O (log − 12 n ) | ℓ ( x ) = 1] ≥ 1 − O ( 1 n 2 ) .

Original: Proof.
Revised: Proof.

Original: We have ∆ i =
Revised: We have ∆ i =

Original: ( β i − β
Revised: ( β i − β

Original: [Equation]
Revised: [Equation]

Original: i )
Revised: i )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We apply the method of averaged bounded difference (Dubhashi & Panconesi, 2009) to estimate ∆ i .
Revised: We apply the method of averaged bounded difference (Dubhashi & Panconesi, 2009) to estimate ∆ i .

Original: In different parameter regimes, ϕ ( α i t y 

Revised: In different parameter regimes, ϕ ( α i t y

Original: + α ′ i
Revised: + α ′ i

Original: t y 1 ) has different activation patterns.
Revised: t y 1 ) has different activation patterns.

Original: In ( G 1 ) and ( G 2 , 1 ), ϕ ( α i t y 

Revised: In ( G 1 ) and ( G 2 , 1 ), ϕ ( α i t y

Original: + α ′ i
Revised: + α ′ i

Original: t y 1 ) is active with probability 1 − O ( 1 n 2 ) for both ℓ ( y )
Revised: t y 1 ) is active with probability 1 − O ( 1 n 2 ) for both ℓ ( y )

Original: = 0 and ℓ ( y )
Revised: = 0 and ℓ ( y )

Original: = 1 .
Revised: = 1 .

Original: For ℓ ( x ) = 0 , at first we estimate E [∆ i ] .
Revised: For ℓ ( x ) = 0 , at first we estimate E [∆ i ] .

Original: By condition (Cond):
Revised: By condition (Cond):

Original: [Equation]
Revised: [Equation]

Original: Let Y j = 4 ϕ
Revised: Let Y j = 4 ϕ

Original: ( α i t yj 

Revised: ( α i t yj

Original: + α ′ i
Revised: + α ′ i

Original: t yj 1 )
Revised: t yj 1 )

Original: n 2 ( p + q ), then ∆ i =
Revised: n 2 ( p + q ), then ∆ i =

Original: ( β i − β ′ i ) (cid:80) j Y j .
Revised: ( β i − β ′ i ) (cid:80) j Y j .

Original: Based on condition (Cond), (cid:12)(cid:12)
Revised: Based on condition (Cond), (cid:12)(cid:12)

Original: Y j − 2 λ ( pα i + qα ′ i ) n ( p + q ) 2 (cid:12)(cid:12) ≤
Revised: Y j − 2 λ ( pα i + qα ′ i ) n ( p + q ) 2 (cid:12)(cid:12) ≤

Original: ( α i − α ′ i )
Revised: ( α i − α ′ i )

Original: O (log − 72 n ) for ℓ ( y j ) = 0 .
Revised: O (log − 72 n ) for ℓ ( y j ) = 0 .

Original: For any a k , a ′ k , (cid:12)(cid:12)(cid:12)(cid:12) E
Revised: For any a k , a ′ k , (cid:12)(cid:12)(cid:12)(cid:12) E

Original: [ Y k | Y 1 , ·
Revised: [ Y k | Y 1 , ·

Original: [Equation]
Revised: [Equation]

Original: · , Y k −
Revised: · , Y k −

Original: [Equation]
Revised: [Equation]

Original: , Y k =
Revised: , Y k =

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: Y k | Y
Revised: Y k | Y

Original: [Equation]
Revised: [Equation]

Original: , Y k
Revised: , Y k

Original: [Equation]
Revised: [Equation]

Original: , Y k
Revised: , Y k

Original: [Equation]
Revised: [Equation]

Original: Moreover, when the number of vertices with revealed labels are fixed,
Revised: Moreover, when the number of vertices with revealed labels are fixed,

Original: [Equation]
Revised: [Equation]

Original: for j ≥ k .
Revised: for j ≥ k .

Original: By condition (Cond)
Revised: By condition (Cond)

Original: [Equation]
Revised: [Equation]

Original: there are at most O (log 3 n ) non-zero terms for Y k , 1 ≤ k ≤ n .
Revised: there are at most O (log 3 n ) non-zero terms for Y k , 1 ≤ k ≤ n .

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: for 1 ≤ k ≤ n .
Revised: for 1 ≤ k ≤ n .

Original: By the method of averaged bounded difference
Revised: By the method of averaged bounded difference

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Other regimes can be proved similarly
Revised: Other regimes can be proved similarly

Original: [Equation]
Revised: [Equation]

Original: A. “B AD T YPE ” N EURONS
Revised: A. “B AD T YPE ” N EURONS

Original: For convenience of our analysis
Revised: For convenience of our analysis

Original: [Equation]
Revised: [Equation]

Original: we further divide ( B 2 ) into subcases ( B 2 , 1 ), ( B 2 , 2 ) and ( B 2 , 3 ) according to the ratio of
Revised: we further divide ( B 2 ) into subcases ( B 2 , 1 ), ( B 2 , 2 ) and ( B 2 , 3 ) according to the ratio of

Original: [Equation]
Revised: [Equation]

Original: α i α ′ i
Revised: α i α ′ i

Original: [Equation]
Revised: [Equation]

Original: We have the following estimation of ∆ i in “bad type”
Revised: We have the following estimation of ∆ i in “bad type”

Original: [Equation]
Revised: [Equation]

Original: Theorem A.2 (concentration of output from “bad type” neurons) .
Revised: Theorem A.2 (concentration of output from “bad type” neurons) .

Original: If the i -th neuron is of “bad type”, we have:
Revised: If the i -th neuron is of “bad type”, we have:

Original: • in ( B 1 ) and ( B 2 , 1 ):
Revised: • in ( B 1 ) and ( B 2 , 1 ):

Original: [Equation]
Revised: [Equation]

Original: • in ( B 2 , 2 ):
Revised: • in ( B 2 , 2 ):

Original: [Equation]
Revised: [Equation]

Original: • in ( B 2 , 3 ):
Revised: • in ( B 2 , 3 ):

Original: [Equation]
Revised: [Equation]

Original: • in ( B 3 ):
Revised: • in ( B 3 ):

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is similar as Theorem A
Revised: The proof is similar as Theorem A

Original: [Equation]
Revised: [Equation]

Original: A. 
Revised: A.

Original: “H ARMLESS T
Revised: “H ARMLESS T

Original: YPE ” N EURONS
Revised: YPE ” N EURONS

Original: We have the following estimation of ∆ i in “harmless type”.
Revised: We have the following estimation of ∆ i in “harmless type”.

Original: Theorem A.3 (concentration of output from “harmless type” neurons) .
Revised: Theorem A.3 (concentration of output from “harmless type” neurons) .

Original: If the i -th neuron is of “harmless type”, we have:
Revised: If the i -th neuron is of “harmless type”, we have:

Original: • in ( H 1 ):
Revised: • in ( H 1 ):

Original: [Equation]
Revised: [Equation]

Original: • in ( H 2 ): ∆ i = 0 for both ℓ ( x ) = 0 and 1.
Revised: • in ( H 2 ): ∆ i = 0 for both ℓ ( x ) = 0 and 1.

Original: A. S EPARATION OF O UTPUT
Revised: A. S EPARATION OF O UTPUT

Original: Previous subsections have shown the concentration of ∆ i for each type of neurons.
Revised: Previous subsections have shown the concentration of ∆ i for each type of neurons.

Original: For the i -th neuron, we write the concentrated value as m i 

Revised: For the i -th neuron, we write the concentrated value as m i

Original: if ℓ ( x ) = 

Revised: if ℓ ( x ) =

Original: and m i 1 if ℓ ( x ) = 1 .
Revised: and m i 1 if ℓ ( x ) = 1 .

Original: From Theorem
Revised: From Theorem

Original: A.1, A.2 and A.3, we have the following result about the value of m i 0 − m i 1 by straightforward computation.
Revised: A.1, A.2 and A.3, we have the following result about the value of m i 0 − m i 1 by straightforward computation.

Original: Corollary A.
Revised: Corollary A.

Original: [Equation]
Revised: [Equation]

Original: We have the following result about m i 0 − m i 1 for 1 ≤ i ≤ h :
Revised: We have the following result about m i 0 − m i 1 for 1 ≤ i ≤ h :

Original: • if the i -the neuron is of “good type”: in ( G 1 ) and ( G 2 , 1 ) : m i 0 − m i 1 = λ | α
Revised: • if the i -the neuron is of “good type”: in ( G 1 ) and ( G 2 , 1 ) : m i 0 − m i 1 = λ | α

Original: i − α ′
Revised: i − α ′

Original: [Equation]
Revised: [Equation]

Original: where Λ 3 = (1 − log − 13 n ) p 2 − q
Revised: where Λ 3 = (1 − log − 13 n ) p 2 − q

Original: [Equation]
Revised: [Equation]

Original: • if the i -the neuron is of “bad type”: in ( B 1 ) and ( B 2 , 1 ) : m i
Revised: • if the i -the neuron is of “bad type”: in ( B 1 ) and ( B 2 , 1 ) : m i

Original: [Equation]
Revised: [Equation]

Original: where Λ 1 = (1+log − 13 n ) p 2 − q(1+log − 13 n ) p + q
Revised: where Λ 1 = (1+log − 13 n ) p 2 − q(1+log − 13 n ) p + q

Original: [Equation]
Revised: [Equation]

Original: Λ 3 = (
Revised: Λ 3 = (

Original: [Equation]
Revised: [Equation]

Original: log
Revised: log

Original: [Equation]
Revised: [Equation]

Original: • if the i -the neuron is of “harmless type”:
Revised: • if the i -the neuron is of “harmless type”:

Original: [Equation]
Revised: [Equation]

Original: where Λ 5 = pq log −
Revised: where Λ 5 = pq log −

Original: [Equation]
Revised: [Equation]

Original: B D YNAMICS OF P ARAMETERS
Revised: B D YNAMICS OF P ARAMETERS

Original: We consider the cross-entropy loss in training.
Revised: We consider the cross-entropy loss in training.

Original: The loss on a particular vertex x is
Revised: The loss on a particular vertex x is

Original: [Equation]
Revised: [Equation]

Original: where O 0 ( x ) and O 1 ( x ) are the first and second component of the output respectively, i.e.
Revised: where O 0 ( x ) and O 1 ( x ) are the first and second component of the output respectively, i.e.

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: For a given graph G generated by SBM, we set the objective function L ( G ) as the average loss over all the vertices with revealed labels 2 , i.e.
Revised: For a given graph G generated by SBM, we set the objective function L ( G ) as the average loss over all the vertices with revealed labels 2 , i.e.

Original: L ( G ) =
Revised: L ( G ) =

Original: # { x ∈ G : ℓ ( x ) is revealed } (cid:88) x : ℓ ( x ) revealed
Revised: # { x ∈ G : ℓ ( x ) is revealed } (cid:88) x : ℓ ( x ) revealed

Original: L ( x ) .
Revised: L ( x ) .

Original: We first show the partial derivatives of parameters.
Revised: We first show the partial derivatives of parameters.

Original: Theorem B.1 (derivatives of parameters) .
Revised: Theorem B.1 (derivatives of parameters) .

Original: For 1 ≤ i ≤ h , let x be a vertex, ℓ ( x ) its true label,
Revised: For 1 ≤ i ≤ h , let x be a vertex, ℓ ( x ) its true label,

Original: L ( x ) = − log O ℓ ( x ) ( x ) , then
Revised: L ( x ) = − log O ℓ ( x ) ( x ) , then

Original: [Equation]
Revised: [Equation]

Original: where Z = exp( g 1 − ℓ ( x ) ( x )) exp( g 0 ( x ))+exp( g 1 ( x )) , t y 0 and t y 1 are the numbers of neighbors of y (including perhaps y itself) with revealed labels in class 0 and class 1 respectively.
Revised: where Z = exp( g 1 − ℓ ( x ) ( x )) exp( g 0 ( x ))+exp( g 1 ( x )) , t y 0 and t y 1 are the numbers of neighbors of y (including perhaps y itself) with revealed labels in class 0 and class 1 respectively.

Original: Proof.
Revised: Proof.

Original: We compute ∂L∂α i , ∂L∂β i and ∂L∂b 0 , others can be computed symmetrically.
Revised: We compute ∂L∂α i , ∂L∂β i and ∂L∂b 0 , others can be computed symmetrically.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: since O j ( x ) = exp( g j ( x )) exp( g 0 ( x ))
Revised: since O j ( x ) = exp( g j ( x )) exp( g 0 ( x ))

Original: [Equation]
Revised: [Equation]

Original: exp
Revised: exp

Original: [Equation]
Revised: [Equation]

Original: ( x ))
Revised: ( x ))

Original: [Equation]
Revised: [Equation]

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: Since g j ( x ) =
Revised: Since g j ( x ) =

Original: f j ( x ) + b j , ∂g j ( x ) ∂α i =
Revised: f j ( x ) + b j , ∂g j ( x ) ∂α i =

Original: ∂f j ( x ) ∂α i , j = 0 , 1 .
Revised: ∂f j ( x ) ∂α i , j = 0 , 1 .

Original: By (2)
Revised: By (2)

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: i 1 [
Revised: i 1 [

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: t y
Revised: t y

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: Therefore ∂L ∂α
Revised: Therefore ∂L ∂α

Original: [Equation]
Revised: [Equation]

Original: Next we compute ∂L∂β i .
Revised: Next we compute ∂L∂β i .

Original: Similar as above, ∂L∂β
Revised: Similar as above, ∂L∂β

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: Lastly, ∂L
Revised: Lastly, ∂L

Original: [Equation]
Revised: [Equation]

Original: since ∂g 0 ( x ) ∂b= 1 , ∂g 1 ( x ) ∂b= 0 .
Revised: since ∂g 0 ( x ) ∂b= 1 , ∂g 1 ( x ) ∂b= 0 .

Original: In the following, we will use Theorem B.1 to analyze the dynamics of neurons of each type.
Revised: In the following, we will use Theorem B.1 to analyze the dynamics of neurons of each type.

Original: As we can see, all of ∂L∂α i , ∂L∂α ′ i , ∂L∂β i and ∂L∂β
Revised: As we can see, all of ∂L∂α i , ∂L∂α ′ i , ∂L∂β i and ∂L∂β

Original: i have the form Y Z .
Revised: i have the form Y Z .

Original: In order to estimate these derivatives, we show the concentration of Y and Z respectively.
Revised: In order to estimate these derivatives, we show the concentration of Y and Z respectively.

Original: To estimate the concentration of Z , we need the concentration of output obtained in Section 4.
Revised: To estimate the concentration of Z , we need the concentration of output obtained in Section 4.

Original: For any ϵ > 0 , we require the probability of concentration in (3) to be at least 1 − ˜ ϵ , where ˜ ϵ
Revised: For any ϵ > 0 , we require the probability of concentration in (3) to be at least 1 − ˜ ϵ , where ˜ ϵ

Original: = o ( ϵ ) .
Revised: = o ( ϵ ) .

Original: In particular, if we choose ˜ ϵ
Revised: In particular, if we choose ˜ ϵ

Original: = ϵ 2 , then we set 1 − O ( 1 n ) ≥ 1 − ϵ 2 , i.e.
Revised: = ϵ 2 , then we set 1 − O ( 1 n ) ≥ 1 − ϵ 2 , i.e.

Original: [Equation]
Revised: [Equation]

Original: Our following analysis will be based on this condition.
Revised: Our following analysis will be based on this condition.

Original: Meanwhile in order to have balanced performance in each epoch of coordinate descent, we require (cid:12)(cid:12) E
Revised: Meanwhile in order to have balanced performance in each epoch of coordinate descent, we require (cid:12)(cid:12) E

Original: [ ∂L∂b 0 ] (cid:12)(cid:12) <
Revised: [ ∂L∂b 0 ] (cid:12)(cid:12) <

Original: ˜ ϵ 2 .
Revised: ˜ ϵ 2 .

Original: Since E [ ∂L∂b 0 ] = 12 ( − E
Revised: Since E [ ∂L∂b 0 ] = 12 ( − E

Original: [ Z | ℓ ( x ) = 0]
Revised: [ Z | ℓ ( x ) = 0]

Original: + E [ Z | ℓ ( x ) = 1])) , we have
Revised: + E [ Z | ℓ ( x ) = 1])) , we have

Original: [Equation]
Revised: [Equation]

Original: We have the following relation between µ 0 and µ 1 .
Revised: We have the following relation between µ 0 and µ 1 .

Original: In the following, σ represents the sigmoid function: σ ( x ) = 1 1+ e − x .
Revised: In the following, σ represents the sigmoid function: σ ( x ) = 1 1+ e − x .

Original: Proposition B.2.
Revised: Proposition B.2.

Original: If (cid:12)(cid:12) E
Revised: If (cid:12)(cid:12) E

Original: [ Z | ℓ ( x ) = 0] − E
Revised: [ Z | ℓ ( x ) = 0] − E

Original: [ Z | ℓ ( x ) = 1] (cid:12)(cid:12) <
Revised: [ Z | ℓ ( x ) = 1] (cid:12)(cid:12) <

Original: ˜ ϵ , then | σ ( − µ 0 ) − σ ( µ 1 ) | ≤ σ ′ ( µ 0 − δ )
Revised: ˜ ϵ , then | σ ( − µ 0 ) − σ ( µ 1 ) | ≤ σ ′ ( µ 0 − δ )

Original: δ + σ ′ ( µ 1 + δ ) δ
Revised: δ + σ ′ ( µ 1 + δ ) δ

Original: + 3˜ ϵ , where δ is as shown in Corollary 4.2.
Revised: + 3˜ ϵ , where δ is as shown in Corollary 4.2.

Original: Proof.
Revised: Proof.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: For ℓ ( x ) = 0 , by Lagrange mean value theorem, | σ ( − µ 0 ) − Z | = | σ ( − µ 0 ) − σ ( − ∆) | = σ ′ ( ξ )(∆ − µ 0 ) , where ξ is between − µ 0 and − ∆ .
Revised: For ℓ ( x ) = 0 , by Lagrange mean value theorem, | σ ( − µ 0 ) − Z | = | σ ( − µ 0 ) − σ ( − ∆) | = σ ′ ( ξ )(∆ − µ 0 ) , where ξ is between − µ 0 and − ∆ .

Original: By Corollary 4.2 and the condition of n , | ∆ − µ 0 | ≤ δ with probability ≥ 1 − ˜ ϵ .
Revised: By Corollary 4.2 and the condition of n , | ∆ − µ 0 | ≤ δ with probability ≥ 1 − ˜ ϵ .

Original: From the remark following Corollary A.4, we have σ ′ ( ξ ) ≤ σ ′ ( − µ 0 + δ ) =
Revised: From the remark following Corollary A.4, we have σ ′ ( ξ ) ≤ σ ′ ( − µ 0 + δ ) =

Original: σ ′ ( µ 0 − δ ) , 3 with probability ≥ 1 − ˜ ϵ .
Revised: σ ′ ( µ 0 − δ ) , 3 with probability ≥ 1 − ˜ ϵ .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: then (note that 0 <
Revised: then (note that 0 <

Original: Z < 1 )
Revised: Z < 1 )

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e.
Revised: e.

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: By triangle inequality,
Revised: By triangle inequality,

Original: [Equation]
Revised: [Equation]

Original: From the proof above, we can directly obtain the following corollary about Z .
Revised: From the proof above, we can directly obtain the following corollary about Z .

Original: Corollary B.3.
Revised: Corollary B.3.

Original: [Equation]
Revised: [Equation]

Original: In order to obtain the concentration of Z , we need to estimate σ ′ ( µ 0 − δ ) δ and σ ′ ( µ 1 + δ ) δ .
Revised: In order to obtain the concentration of Z , we need to estimate σ ′ ( µ 0 − δ ) δ and σ ′ ( µ 1 + δ ) δ .

Original: The following proposition is based on the condition that | µ 0 + µ 1 | ≥ 4 δ .
Revised: The following proposition is based on the condition that | µ 0 + µ 1 | ≥ 4 δ .

Original: If | µ 0 + µ 1 | < 4 δ , set c := µ 0 − µ 1 , we have µ 0 > c 2 − 2 δ and µ 1 < − c 2 + 2 δ .
Revised: If | µ 0 + µ 1 | < 4 δ , set c := µ 0 − µ 1 , we have µ 0 > c 2 − 2 δ and µ 1 < − c 2 + 2 δ .

Original: Then the concentration of output shown in
Revised: Then the concentration of output shown in

Original: Corollary 4.2 can guarantee 1 − ϵ accuracy of the model for any ϵ > 0 .
Revised: Corollary 4.2 can guarantee 1 − ϵ accuracy of the model for any ϵ > 0 .

Original: In fact, from | ∆ − µ 0 | < δ , we have ∆ > µ 0 − δ > c 2 − 3 δ > 0 , due to δ =
Revised: In fact, from | ∆ − µ 0 | < δ , we have ∆ > µ 0 − δ > c 2 − 3 δ > 0 , due to δ =

Original: o ( c ) .
Revised: o ( c ) .

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: P [∆ <
Revised: P [∆ <

Original: [Equation]
Revised: [Equation]

Original: ] ≥
Revised: ] ≥

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ϵ.
Revised: ϵ.

Original: Since ˜ ϵ
Revised: Since ˜ ϵ

Original: = o ( ϵ ) , the model achieves overall accuracy ≥ 1 − ϵ .
Revised: = o ( ϵ ) , the model achieves overall accuracy ≥ 1 − ϵ .

Original: Proposition B.4.
Revised: Proposition B.4.

Original: If | µ 0 + µ 1 | ≥ 4 δ , then σ ′ ( µ 0 − δ )
Revised: If | µ 0 + µ 1 | ≥ 4 δ , then σ ′ ( µ 0 − δ )

Original: δ
Revised: δ

Original: = O (˜ ϵ ) , σ ′ ( µ 1 + δ ) δ
Revised: = O (˜ ϵ ) , σ ′ ( µ 1 + δ ) δ

Original: = O (˜ ϵ ) .
Revised: = O (˜ ϵ ) .

Original: Proof.
Revised: Proof.

Original: First, we estimate the lower bound of | σ ( − µ 0 ) − σ ( µ 1 ) | via the Fundamental Theorem of
Revised: First, we estimate the lower bound of | σ ( − µ 0 ) − σ ( µ 1 ) | via the Fundamental Theorem of

Original: Calculus.
Revised: Calculus.

Original: We have | σ ( − µ 0 ) − σ
Revised: We have | σ ( − µ 0 ) − σ

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: | =
Revised: | =

Original: [Equation]
Revised: [Equation]

Original: dt
Revised: dt

Original: [Equation]
Revised: [Equation]

Original: If − µ 0 < µ 1 < 0 , since µ 0 + µ 1 ≥ 4 δ , we divide the interval [ − µ 0 , µ 1 ] into [ − µ 0 , − µ 0 + 2 δ ] ∪ [ − µ 

Revised: If − µ 0 < µ 1 < 0 , since µ 0 + µ 1 ≥ 4 δ , we divide the interval [ − µ 0 , µ 1 ] into [ − µ 0 , − µ 0 + 2 δ ] ∪ [ − µ

Original: + 2 δ, µ 1 − 2 δ ] ∪ [ µ 1 − 2 δ, µ 1 ] and estimate the lower bound of the integral.
Revised: + 2 δ, µ 1 − 2 δ ] ∪ [ µ 1 − 2 δ, µ 1 ] and estimate the lower bound of the integral.

Original: Since σ ′ ( x ) is increasing on ( −∞ , 0] , we have
Revised: Since σ ′ ( x ) is increasing on ( −∞ , 0] , we have

Original: [Equation]
Revised: [Equation]

Original: where I 1 = (cid:82) µ 1 − 2 δ − µ 

Revised: where I 1 = (cid:82) µ 1 − 2 δ − µ

Original: +2 δ σ ′ ( t ) dt .
Revised: +2 δ σ ′ ( t ) dt .

Original: If µ 1 < − µ 0 < 0 , similarly we have
Revised: If µ 1 < − µ 0 < 0 , similarly we have

Original: [Equation]
Revised: [Equation]

Original: where I 2 = (cid:82) − µ 0 −
Revised: where I 2 = (cid:82) − µ 0 −

Original: [Equation]
Revised: [Equation]

Original: 1 +2 δ σ ′ ( t ) dt .
Revised: 1 +2 δ σ ′ ( t ) dt .

Original: We have a uniform lower bound from (7) and (8):
Revised: We have a uniform lower bound from (7) and (8):

Original: [Equation]
Revised: [Equation]

Original: where I = min { I 1 , I 2 } .
Revised: where I = min { I 1 , I 2 } .

Original: Furthermore, by Proposition B.2,
Revised: Furthermore, by Proposition B.2,

Original: [Equation]
Revised: [Equation]

Original: Combine (9) and (10):
Revised: Combine (9) and (10):

Original: [Equation]
Revised: [Equation]

Original: By Lagrange mean value theorem,
Revised: By Lagrange mean value theorem,

Original: [Equation]
Revised: [Equation]

Original: where ξ 0 ∈ ( − µ 0 − 2 δ, − µ 0 + δ ) , ξ 1 ∈ ( µ 1 − 2 δ, µ 1 + δ )
Revised: where ξ 0 ∈ ( − µ 0 − 2 δ, − µ 0 + δ ) , ξ 1 ∈ ( µ 1 − 2 δ, µ 1 + δ )

Original: Plug these into (11):
Revised: Plug these into (11):

Original: [Equation]
Revised: [Equation]

Original: Since δ 2 ( σ ′′ ( ξ 0 ) + σ ′′ ( ξ 1 ))
Revised: Since δ 2 ( σ ′′ ( ξ 0 ) + σ ′′ ( ξ 1 ))

Original: =
Revised: =

Original: o ( σ ′ ( µ 0 − δ ) δ ) and o ( σ ′ ( µ 1 + δ ) δ ) , we have
Revised: o ( σ ′ ( µ 0 − δ ) δ ) and o ( σ ′ ( µ 1 + δ ) δ ) , we have

Original: [Equation]
Revised: [Equation]

Original: = O
Revised: = O

Original: (˜
Revised: (˜

Original: [Equation]
Revised: [Equation]

Original: Combine Proposition B.4 and Corollary B.3, we have the following concentration of Z .
Revised: Combine Proposition B.4 and Corollary B.3, we have the following concentration of Z .

Original: Proposition B.5.
Revised: Proposition B.5.

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: Under the condition of balanced performance, we have the following corollary about the concentration of Z independent of the label of x .
Revised: Under the condition of balanced performance, we have the following corollary about the concentration of Z independent of the label of x .

Original: Corollary B.6.
Revised: Corollary B.6.

Original: If (cid:12)(cid:12) E
Revised: If (cid:12)(cid:12) E

Original: [ ∂L∂b 0 ] (cid:12)(cid:12) ≤ ˜ ϵ 2 , then P [ (cid:12)(cid:12)
Revised: [ ∂L∂b 0 ] (cid:12)(cid:12) ≤ ˜ ϵ 2 , then P [ (cid:12)(cid:12)

Original: Z − E
Revised: Z − E

Original: [ Z ] (cid:12)(cid:12) ≤ O (˜ ϵ )] ≥ 1 − ˜ ϵ .
Revised: [ Z ] (cid:12)(cid:12) ≤ O (˜ ϵ )] ≥ 1 − ˜ ϵ .

Original: Proof.
Revised: Proof.

Original: Since E [ ∂L∂b 0 ] = 12 ( E [ Z | ℓ ( x ) = 0] − E [ Z | ℓ ( x ) = 1]) , we have
Revised: Since E [ ∂L∂b 0 ] = 12 ( E [ Z | ℓ ( x ) = 0] − E [ Z | ℓ ( x ) = 1]) , we have

Original: [Equation]
Revised: [Equation]

Original: On the other hand,
Revised: On the other hand,

Original: [Equation]
Revised: [Equation]

Original: So we have (cid:12)(cid:12) E
Revised: So we have (cid:12)(cid:12) E

Original: [ Z ] − E [ Z | ℓ ( x ) = 0] (cid:12)(cid:12) ≤ ˜ ϵ 2 .
Revised: [ Z ] − E [ Z | ℓ ( x ) = 0] (cid:12)(cid:12) ≤ ˜ ϵ 2 .

Original: By Proposition B.5, P [ | Z − E
Revised: By Proposition B.5, P [ | Z − E

Original: [ Z ] | ≤ O (˜ ϵ )] ≥ 1 − ˜ ϵ .
Revised: [ Z ] | ≤ O (˜ ϵ )] ≥ 1 − ˜ ϵ .

Original: Now we can derive the estimation of the derivatives.
Revised: Now we can derive the estimation of the derivatives.

Original: Theorem B.7 (concentration of derivatives) .
Revised: Theorem B.7 (concentration of derivatives) .

Original: For loss on the whole graph L =
Revised: For loss on the whole graph L =

Original: L ( G ) , with probability ≥ 1 − O ( 1 n ) , we have
Revised: L ( G ) , with probability ≥ 1 − O ( 1 n ) , we have

Original: If α i > α ′ i > 0 or α i > 0 > α ′ i , | α i α ′ i |
Revised: If α i > α ′ i > 0 or α i > 0 > α ′ i , | α i α ′ i |

Original: ≥
Revised: ≥

Original: pq
Revised: pq

Original: +
Revised: +

Original: log − 13 n ) , then
Revised: log − 13 n ) , then

Original: [Equation]
Revised: [Equation]

Original: If α i > 0 > α ′ i , | α
Revised: If α i > 0 > α ′ i , | α

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ′ i | ∈
Revised: ′ i | ∈

Original: [ qp (1 + γ ) , pq (1 − log − 13 n )] , where γ ∈ [log − 13 n, ( pq )
Revised: [ qp (1 + γ ) , pq (1 − log − 13 n )] , where γ ∈ [log − 13 n, ( pq )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: log − 13 n ) − 1] , then
Revised: log − 13 n ) − 1] , then

Original: [Equation]
Revised: [Equation]

Original: If α i > 0 > α ′ i , | α
Revised: If α i > 0 > α ′ i , | α

Original: i α ′ i | ∈ ( pq
Revised: i α ′ i | ∈ ( pq

Original: [Equation]
Revised: [Equation]

Original: 1 − log − 13 n ) , pq
Revised: 1 − log − 13 n ) , pq

Original: [Equation]
Revised: [Equation]

Original: 1 + log − 13 n )) , then
Revised: 1 + log − 13 n )) , then

Original: [Equation]
Revised: [Equation]

Original: Λ
Revised: Λ

Original: [Equation]
Revised: [Equation]

Original: • if β i > β ′ i ,
Revised: • if β i > β ′ i ,

Original: [Equation]
Revised: [Equation]

Original: • if β
Revised: • if β

Original: i ≤ β ′ i , ∂L ∂α i
Revised: i ≤ β ′ i , ∂L ∂α i

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [ Z ] (cid:18)
Revised: [ Z ] (cid:18)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ] (22) ∂L ∂α ′
Revised: ] (22) ∂L ∂α ′

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ] (23) ∂L ∂α i
Revised: ] (23) ∂L ∂α i

Original: [Equation]
Revised: [Equation]

Original: ∂L
Revised: ∂L

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Z ]
Revised: Z ]

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: O (
Revised: O (

Original: [Equation]
Revised: [Equation]

Original: 12 n ) (cid:19) ] (24)
Revised: 12 n ) (cid:19) ] (24)

Original: If α i > 0 > α ′ i , (cid:12)(cid:12) α
Revised: If α i > 0 > α ′ i , (cid:12)(cid:12) α

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12) ≤ qp (1 + log − 13 n ) , β i <
Revised: (cid:12)(cid:12) ≤ qp (1 + log − 13 n ) , β i <

Original: β ′ i , then
Revised: β ′ i , then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We show the proof for item 1, other items can be proved similarly.
Revised: We show the proof for item 1, other items can be proved similarly.

Original: Since L ( G ) is the average of the losses over revealed vertices, we first show the concentration of ∂L ( x ) ∂α i , then we show the concentration of ∂L ( G ) ∂α i using union bound.
Revised: Since L ( G ) is the average of the losses over revealed vertices, we first show the concentration of ∂L ( x ) ∂α i , then we show the concentration of ∂L ( G ) ∂α i using union bound.

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: we first show the concentration of Y
Revised: we first show the concentration of Y

Original: [Equation]
Revised: [Equation]

Original: using the method of averaged bounded difference.
Revised: using the method of averaged bounded difference.

Original: Similar as the proof of Theorem A.1, let Y j =
Revised: Similar as the proof of Theorem A.1, let Y j =

Original: [Equation]
Revised: [Equation]

Original: Based on Condition (Cond), for ℓ ( x ) = 0 , | Y j + 2 λp n ( p + q
Revised: Based on Condition (Cond), for ℓ ( x ) = 0 , | Y j + 2 λp n ( p + q

Original: [Equation]
Revised: [Equation]

Original: 2 | ≤
Revised: 2 | ≤

Original: O (log − 72 n ) for ℓ ( y j ) = 0 .
Revised: O (log − 72 n ) for ℓ ( y j ) = 0 .

Original: Similar results hold for ℓ ( y j )
Revised: Similar results hold for ℓ ( y j )

Original: = 1 , ℓ ( x ) = 1 .
Revised: = 1 , ℓ ( x ) = 1 .

Original: So for any a k , a ′ k ,
Revised: So for any a k , a ′ k ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , · · · , Y k − 1 , Y k = a k ] − E [ (cid:88)
Revised: , · · · , Y k − 1 , Y k = a k ] − E [ (cid:88)

Original: j
Revised: j

Original: Y j | Y 1 , · · · , Y
Revised: Y j | Y 1 , · · · , Y

Original: [Equation]
Revised: [Equation]

Original: − 1 , Y
Revised: − 1 , Y

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ] (cid:12)(cid:12)(cid:12)(cid:12) ≤ (
Revised: ] (cid:12)(cid:12)(cid:12)(cid:12) ≤ (

Original: [Equation]
Revised: [Equation]

Original: − α ′
Revised: − α ′

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: O (log − 72 n ) .
Revised: O (log − 72 n ) .

Original: By method of averaged bounded difference, for ℓ ( x ) = 0 ,
Revised: By method of averaged bounded difference, for ℓ ( x ) = 0 ,

Original: P [ (cid:12)(cid:12)(cid:12)(cid:12) (cid:88) ℓ ( y j )=
Revised: P [ (cid:12)(cid:12)(cid:12)(cid:12) (cid:88) ℓ ( y j )=

Original: Y j
Revised: Y j

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: (cid:0)
Revised: (cid:0)

Original: [Equation]
Revised: [Equation]

Original: (cid:1)
Revised: (cid:1)

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12) ≤
Revised: (cid:12)(cid:12)(cid:12)(cid:12) ≤

Original: O (log − 12 n )] ≥ 1 − exp ( − 2 log 3 n ) ≥ 1 − 1 n 2 .
Revised: O (log − 12 n )] ≥ 1 − exp ( − 2 log 3 n ) ≥ 1 − 1 n 2 .

Original: Similarly
Revised: Similarly

Original: P [ (cid:12)(cid:12)(cid:12)(cid:12) (cid:88) ℓ ( y j )=
Revised: P [ (cid:12)(cid:12)(cid:12)(cid:12) (cid:88) ℓ ( y j )=

Original: Y j + λ (cid:0) q p + q (cid:1) 2 (cid:12)(cid:12)(cid:12)(cid:12) ≤
Revised: Y j + λ (cid:0) q p + q (cid:1) 2 (cid:12)(cid:12)(cid:12)(cid:12) ≤

Original: O (log − 12 n )] ≥ 1 − 1 n 2 .
Revised: O (log − 12 n )] ≥ 1 − 1 n 2 .

Original: Hence
Revised: Hence

Original: P [ (cid:12)(cid:12)(cid:12)(cid:12) Y + λ ( p 2 + q 2 ) ( p + q ) 2 (cid:12)(cid:12)(cid:12)(cid:12) ≤ O (log − 12 n )] ≥ 1 − 1 n 2 .
Revised: P [ (cid:12)(cid:12)(cid:12)(cid:12) Y + λ ( p 2 + q 2 ) ( p + q ) 2 (cid:12)(cid:12)(cid:12)(cid:12) ≤ O (log − 12 n )] ≥ 1 − 1 n 2 .

Original: By Corollary B.6, P [ | Z − E
Revised: By Corollary B.6, P [ | Z − E

Original: [ Z ] | ≤ O (˜ ϵ )] ≥ 1 − ˜ ϵ , so we have
Revised: [ Z ] | ≤ O (˜ ϵ )] ≥ 1 − ˜ ϵ , so we have

Original: P [ (cid:12)(cid:12)(cid:12)(cid:12) ∂L ( x ) ∂α i + ( β i − β ′ i )
Revised: P [ (cid:12)(cid:12)(cid:12)(cid:12) ∂L ( x ) ∂α i + ( β i − β ′ i )

Original: λ p 2 + q( p + q ) 2 E
Revised: λ p 2 + q( p + q ) 2 E

Original: [ Z ] (cid:12)(cid:12)(cid:12)(cid:12) ≤ | β
Revised: [ Z ] (cid:12)(cid:12)(cid:12)(cid:12) ≤ | β

Original: i − β ′ i | E
Revised: i − β ′ i | E

Original: [ Z ] O (log −
Revised: [ Z ] O (log −

Original: [Equation]
Revised: [Equation]

Original: For ℓ ( x ) = 1 , similarly we have
Revised: For ℓ ( x ) = 1 , similarly we have

Original: [Equation]
Revised: [Equation]

Original: By union bound, we have (12).
Revised: By union bound, we have (12).

Original: (13) and (14) can be proved similarly.
Revised: (13) and (14) can be proved similarly.

Original: Using Theorem B.7, we can analyze dynamics of neurons of each type.
Revised: Using Theorem B.7, we can analyze dynamics of neurons of each type.

Original: First, we introduce some notations.
Revised: First, we introduce some notations.

Original: Let η k denote the learning rate at the k -th epoch, Z ( k ) be the value of Z at the k -th epoch, α ( k ) i be the value of α i at the k -th epoch, similar for α ′ ( k ) i , β ( k ) i and β ′ ( k ) i .
Revised: Let η k denote the learning rate at the k -th epoch, Z ( k ) be the value of Z at the k -th epoch, α ( k ) i be the value of α i at the k -th epoch, similar for α ′ ( k ) i , β ( k ) i and β ′ ( k ) i .

Original: In particular, α (0) i , α ′ (0) i , β (0) i and β ′ (0) i represent the values at initialization.
Revised: In particular, α (0) i , α ′ (0) i , β (0) i and β ′ (0) i represent the values at initialization.

Original: B. “G OOD T YPE ” N EURONS
Revised: B. “G OOD T YPE ” N EURONS

Original: In this section, we show that “good type” neurons stay in the “good type” regime throughout coordinate descent (Theorem B.8) using Theorem B.7.
Revised: In this section, we show that “good type” neurons stay in the “good type” regime throughout coordinate descent (Theorem B.8) using Theorem B.7.

Original: Theorem B.8.
Revised: Theorem B.8.

Original: “Good type” neurons are preserved in the “good type” throughout coordinate descent with probability ≥ 1 − O ( 1 n 2 ) over the SBM randomness.
Revised: “Good type” neurons are preserved in the “good type” throughout coordinate descent with probability ≥ 1 − O ( 1 n 2 ) over the SBM randomness.

Original: Proof.
Revised: Proof.

Original: As shown in Section 4, “good type” regime is composed of ( G 1 ) and ( G 2 ), we show the dynamics of neurons in ( G 1 ) and ( G 2 ) respectively.
Revised: As shown in Section 4, “good type” regime is composed of ( G 1 ) and ( G 2 ), we show the dynamics of neurons in ( G 1 ) and ( G 2 ) respectively.

Original: Assume that neuron ( α ( k ) i , α ′ ( k ) i , β ( k ) i , β ′ ( k ) i ) is in ( G 1 ), we show that it either stays in ( G 1 ) or moves into ( G 2 ) throughout coordinate descent.
Revised: Assume that neuron ( α ( k ) i , α ′ ( k ) i , β ( k ) i , β ′ ( k ) i ) is in ( G 1 ), we show that it either stays in ( G 1 ) or moves into ( G 2 ) throughout coordinate descent.

Original: In fact, by (14), with probability ≥ 1 − O ( 1 n 2 ) , ∂L ∂β ( k ) i < 0 < ∂L ∂β ′ ( k ) i , so β ( k +1)
Revised: In fact, by (14), with probability ≥ 1 − O ( 1 n 2 ) , ∂L ∂β ( k ) i < 0 < ∂L ∂β ′ ( k ) i , so β ( k +1)

Original: i > β ( k ) i , β ′ ( k +1)
Revised: i > β ( k ) i , β ′ ( k +1)

Original: i < β ′
Revised: i < β ′

Original: ( k ) i and hence β
Revised: ( k ) i and hence β

Original: ( k +1)
Revised: ( k +1)

Original: i − β ′
Revised: i − β ′

Original: ( k +1)
Revised: ( k +1)

Original: i > β ( k ) i − β ′
Revised: i > β ( k ) i − β ′

Original: ( k ) i > 0 .
Revised: ( k ) i > 0 .

Original: By (12) and (13), ∂L ∂α ( k ) i < 0 < ∂L ∂α ′
Revised: By (12) and (13), ∂L ∂α ( k ) i < 0 < ∂L ∂α ′

Original: ( k ) i , so α ( k +1)
Revised: ( k ) i , so α ( k +1)

Original: i > α ( k ) i , α ′ ( k +1)
Revised: i > α ( k ) i , α ′ ( k +1)

Original: i < α ′ ( k ) i .
Revised: i < α ′ ( k ) i .

Original: If α ′ ( k +1) i > 0 , this neuron stays in ( G 1 ).
Revised: If α ′ ( k +1) i > 0 , this neuron stays in ( G 1 ).

Original: If α ′ ( k +1) i < 0 , since (cid:12)(cid:12)(cid:12)(cid:12)
Revised: If α ′ ( k +1) i < 0 , since (cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: i α ′ ( k +1) i (cid:12)(cid:12)(cid:12)(cid:12) =
Revised: i α ′ ( k +1) i (cid:12)(cid:12)(cid:12)(cid:12) =

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: the neuron moves into ( G 2 ).
Revised: the neuron moves into ( G 2 ).

Original: Assume that neuron is in ( G 2 ), we also show that it either moves into ( G 1 ) or stays in ( G 2 ).
Revised: Assume that neuron is in ( G 2 ), we also show that it either moves into ( G 1 ) or stays in ( G 2 ).

Original: As shown in section 3.2, ( G 2 ) =
Revised: As shown in section 3.2, ( G 2 ) =

Original: ( G 2 , 1 ) ∪ ( G 2 , 2 ) ∪ ( G 2 , 3 ).
Revised: ( G 2 , 1 ) ∪ ( G 2 , 2 ) ∪ ( G 2 , 3 ).

Original: If the neuron is in ( G 2 , 1 ), again by (12), (13) and (14), α ( k +1)
Revised: If the neuron is in ( G 2 , 1 ), again by (12), (13) and (14), α ( k +1)

Original: i > α ( k ) i > 0 > α ′ ( k ) i > α ′ ( k +1) i , (cid:12)(cid:12)
Revised: i > α ( k ) i > 0 > α ′ ( k ) i > α ′ ( k +1) i , (cid:12)(cid:12)

Original: α ( k +1)
Revised: α ( k +1)

Original: i α ′
Revised: i α ′

Original: ( k +1)
Revised: ( k +1)

Original: i (cid:12)(cid:12) > 1 , β ( k +1)
Revised: i (cid:12)(cid:12) > 1 , β ( k +1)

Original: i > β ( k ) i > β ′ ( k ) i > β ′ ( k +1)
Revised: i > β ( k ) i > β ′ ( k ) i > β ′ ( k +1)

Original: i , so the neuron stays in G 2 , 2 .
Revised: i , so the neuron stays in G 2 , 2 .

Original: If the neuron is in G 2 , 2 , by (15), (16) and (17), ∂L ∂β ( k ) i < 0 < ∂L ∂β ′ ( k ) i , so β ( k +1) i > β ′ ( k +1)
Revised: If the neuron is in G 2 , 2 , by (15), (16) and (17), ∂L ∂β ( k ) i < 0 < ∂L ∂β ′ ( k ) i , so β ( k +1) i > β ′ ( k +1)

Original: i .
Revised: i .

Original: Also, ∂L ∂α ( k ) i < ∂L
Revised: Also, ∂L ∂α ( k ) i < ∂L

Original: ∂α ′
Revised: ∂α ′

Original: ( k ) i < 0 , so α ( k +1)
Revised: ( k ) i < 0 , so α ( k +1)

Original: i > α ′ ( k +1)
Revised: i > α ′ ( k +1)

Original: i , (cid:12)(cid:12) α
Revised: i , (cid:12)(cid:12) α

Original: ( k +1)
Revised: ( k +1)

Original: i α ′
Revised: i α ′

Original: ( k +1)
Revised: ( k +1)

Original: i (cid:12)(cid:12) > (cid:12)(cid:12) α ( k ) i
Revised: i (cid:12)(cid:12) > (cid:12)(cid:12) α ( k ) i

Original: α ′ ( k ) i (cid:12)(cid:12) > 1 .
Revised: α ′ ( k ) i (cid:12)(cid:12) > 1 .

Original: If α ′ ( k +1) i < 0 , the neuron stays in G 2 .
Revised: If α ′ ( k +1) i < 0 , the neuron stays in G 2 .

Original: If α ′ ( k +1) i > 0 , it moves into G 1 .
Revised: If α ′ ( k +1) i > 0 , it moves into G 1 .

Original: If the neuron is in
Revised: If the neuron is in

Original: G 2 , 3 , by (18) and (21), ∂L ∂β ( k ) i < 0 <
Revised: G 2 , 3 , by (18) and (21), ∂L ∂β ( k ) i < 0 <

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) i , ∂L
Revised: ) i , ∂L

Original: [Equation]
Revised: [Equation]

Original: ( k )
Revised: ( k )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ( k ) i < 0 , so β ( k +1)
Revised: ( k ) i < 0 , so β ( k +1)

Original: i > β ( k ) i > β ′ ( k ) i > β ′ ( k +1)
Revised: i > β ( k ) i > β ′ ( k ) i > β ′ ( k +1)

Original: i , α ( k +1)
Revised: i , α ( k +1)

Original: i − α ′
Revised: i − α ′

Original: ( k +1)
Revised: ( k +1)

Original: i > α ( k ) i − α ′ ( k ) i > 0 .
Revised: i > α ( k ) i − α ′ ( k ) i > 0 .

Original: By (19) and (20), ∂L
Revised: By (19) and (20), ∂L

Original: [Equation]
Revised: [Equation]

Original: ) E [ Z ]
Revised: ) E [ Z ]

Original: [Equation]
Revised: [Equation]

Original: Similar as in ( G 2 , 2 ), if α ′ ( k +1)
Revised: Similar as in ( G 2 , 2 ), if α ′ ( k +1)

Original: i < 0 , the neuron stays in ( G 2 ).
Revised: i < 0 , the neuron stays in ( G 2 ).

Original: If α ′ ( k +1) i > 0 , it moves into ( G 1 ).
Revised: If α ′ ( k +1) i > 0 , it moves into ( G 1 ).

Original: B. “B AD T YPE ” N EURONS
Revised: B. “B AD T YPE ” N EURONS

Original: As shown in Section 4, neurons of “bad type” consist of two cases: B 1 and B 2 , where B 2 = B 2 , 1 ∪
Revised: As shown in Section 4, neurons of “bad type” consist of two cases: B 1 and B 2 , where B 2 = B 2 , 1 ∪

Original: B 2 , 2 ∪ B 2 , 3 ∪ B 3 .
Revised: B 2 , 2 ∪ B 2 , 3 ∪ B 3 .

Original: Since the output in B 3 is concentrated at 0 (see Theorem A.2), we don’t need to worry if neurons move into this region.
Revised: Since the output in B 3 is concentrated at 0 (see Theorem A.2), we don’t need to worry if neurons move into this region.

Original: Neurons in B 1 ∪ B 2 , 1 ∪ B 2 , 2 ∪ B 2 , 3 might exit “bad type” regime and become “harmless” or “good” (if the neuron becomes order-aligned), which will do no harm to the performance of the model.
Revised: Neurons in B 1 ∪ B 2 , 1 ∪ B 2 , 2 ∪ B 2 , 3 might exit “bad type” regime and become “harmless” or “good” (if the neuron becomes order-aligned), which will do no harm to the performance of the model.

Original: If they stay in B 1 ∪ B 2 , 1 ∪ B 2 , 2 ∪ B 2 , 3 , the following theorem shows that the separation m i 0 − m i 1 can be upper bounded by initialization.
Revised: If they stay in B 1 ∪ B 2 , 1 ∪ B 2 , 2 ∪ B 2 , 3 , the following theorem shows that the separation m i 0 − m i 1 can be upper bounded by initialization.

Original: In fact, Theorem
Revised: In fact, Theorem

Original: A.4 shows that m i 0 − m i 1 is proportional to | α
Revised: A.4 shows that m i 0 − m i 1 is proportional to | α

Original: i − α
Revised: i − α

Original: i
Revised: i

Original: ||
Revised: ||

Original: β
Revised: β

Original: i − β ′ i | .
Revised: i − β ′ i | .

Original: The next theorem shows that both | α i − α ′ i | and | β i − β
Revised: The next theorem shows that both | α i − α ′ i | and | β i − β

Original: i | shrink throughout coordinate descent.
Revised: i | shrink throughout coordinate descent.

Original: The worst situation is that the magnitude of | α i − α ′ i | and | β i − β ′ i | of neurons in B 3 increase and move into B 1 or B 2 at certain epoch.
Revised: The worst situation is that the magnitude of | α i − α ′ i | and | β i − β ′ i | of neurons in B 3 increase and move into B 1 or B 2 at certain epoch.

Original: From
Revised: From

Original: Theorem B.7 we see that the magnitude can only increase by a limited rate (we can see this more explicitly in Theorem 6.2).
Revised: Theorem B.7 we see that the magnitude can only increase by a limited rate (we can see this more explicitly in Theorem 6.2).

Original: Theorem B.9.
Revised: Theorem B.9.

Original: If ( α ( k ) i , α ′ ( k ) i , β ( k ) i , β ′ ( k ) i ) is in B 1 ∪ B 2 , 1 ∪ B 2 , 2 ∪ B 2 , 3 then with probability ≥ 1 − O ( 1 n 2 ) over the SBM randomness, (cid:12)(cid:12)
Revised: If ( α ( k ) i , α ′ ( k ) i , β ( k ) i , β ′ ( k ) i ) is in B 1 ∪ B 2 , 1 ∪ B 2 , 2 ∪ B 2 , 3 then with probability ≥ 1 − O ( 1 n 2 ) over the SBM randomness, (cid:12)(cid:12)

Original: α
Revised: α

Original: ( k +1)
Revised: ( k +1)

Original: i − α ′
Revised: i − α ′

Original: ( k +1)
Revised: ( k +1)

Original: i (cid:12)(cid:12) ≤ (cid:12)(cid:12)
Revised: i (cid:12)(cid:12) ≤ (cid:12)(cid:12)

Original: α ( k ) i − α ′ ( k ) i (cid:12)(cid:12) , (cid:12)(cid:12) β
Revised: α ( k ) i − α ′ ( k ) i (cid:12)(cid:12) , (cid:12)(cid:12) β

Original: ( k +1)
Revised: ( k +1)

Original: i − β ′
Revised: i − β ′

Original: ( k +1)
Revised: ( k +1)

Original: i (cid:12)(cid:12) ≤ (cid:12)(cid:12) β ( k ) i − β ′
Revised: i (cid:12)(cid:12) ≤ (cid:12)(cid:12) β ( k ) i − β ′

Original: ( k ) i (cid:12)(cid:12) .
Revised: ( k ) i (cid:12)(cid:12) .

Original: Proof.
Revised: Proof.

Original: In B 1 and B 2 , 1 , by (12) and (13), ∂L ∂α ( k ) i > 0 > ∂L ∂α ′
Revised: In B 1 and B 2 , 1 , by (12) and (13), ∂L ∂α ( k ) i > 0 > ∂L ∂α ′

Original: ( k ) i , then α ( k +1)
Revised: ( k ) i , then α ( k +1)

Original: i < α ( k ) i , α ′ ( k +1)
Revised: i < α ( k ) i , α ′ ( k +1)

Original: i > α ′ ( k ) i , so | α
Revised: i > α ′ ( k ) i , so | α

Original: ( k +1)
Revised: ( k +1)

Original: i − α ′
Revised: i − α ′

Original: ( k +1)
Revised: ( k +1)

Original: i | ≤ | α ( k )
Revised: i | ≤ | α ( k )

Original: i − α ′ ( k ) i | .
Revised: i − α ′ ( k ) i | .

Original: Similarly, by (14), ∂L ∂β ( k ) i = − ∂L ∂β ′ ( k ) i < 0 , so | β ( k +1) i − β ′
Revised: Similarly, by (14), ∂L ∂β ( k ) i = − ∂L ∂β ′ ( k ) i < 0 , so | β ( k +1) i − β ′

Original: ( k +1)
Revised: ( k +1)

Original: i | ≤ | β
Revised: i | ≤ | β

Original: ( k )
Revised: ( k )

Original: i − β ′
Revised: i − β ′

Original: ( k ) i | (Note that α ( k ) i > α ′ ( k ) i , β ( k ) i < β ′ ( k ) i ).
Revised: ( k ) i | (Note that α ( k ) i > α ′ ( k ) i , β ( k ) i < β ′ ( k ) i ).

Original: In B 2 , 2 , from (15) and (16), we have ∂L ∂α ( k ) i > ∂L
Revised: In B 2 , 2 , from (15) and (16), we have ∂L ∂α ( k ) i > ∂L

Original: ∂α ′
Revised: ∂α ′

Original: ( k ) i > 0 , so | α ( k +1)
Revised: ( k ) i > 0 , so | α ( k +1)

Original: i − α ′
Revised: i − α ′

Original: ( k +1)
Revised: ( k +1)

Original: i | ≤ | α ( k )
Revised: i | ≤ | α ( k )

Original: i − α ′ ( k ) i | .
Revised: i − α ′ ( k ) i | .

Original: On the other hand, ∂L ∂β ( k ) i < 0 < ∂L ∂β ′ ( k ) i , so β ( k +1)
Revised: On the other hand, ∂L ∂β ( k ) i < 0 < ∂L ∂β ′ ( k ) i , so β ( k +1)

Original: i > β ( k ) i , β ′ ( k +1)
Revised: i > β ( k ) i , β ′ ( k +1)

Original: i < β ′
Revised: i < β ′

Original: ( k ) i and | β ( k +1)
Revised: ( k ) i and | β ( k +1)

Original: i − β ′
Revised: i − β ′

Original: ( k +1)
Revised: ( k +1)

Original: i | ≤ | β
Revised: i | ≤ | β

Original: ( k )
Revised: ( k )

Original: i − β ′
Revised: i − β ′

Original: ( k ) i | .
Revised: ( k ) i | .

Original: In B 2 , 3 , by (24), ∂L ∂α ( k ) i − ∂L
Revised: In B 2 , 3 , by (24), ∂L ∂α ( k ) i − ∂L

Original: ∂α ′
Revised: ∂α ′

Original: ( k ) i > 0 , so | α ( k +1)
Revised: ( k ) i > 0 , so | α ( k +1)

Original: i − α ′
Revised: i − α ′

Original: ( k +1)
Revised: ( k +1)

Original: i | ≤ | α ( k )
Revised: i | ≤ | α ( k )

Original: i − α ′ ( k ) i | .
Revised: i − α ′ ( k ) i | .

Original: By (18), ∂L ∂β ( k ) i < 0 < ∂L ∂β ′ ( k ) i , so | β ( k +1)
Revised: By (18), ∂L ∂β ( k ) i < 0 < ∂L ∂β ′ ( k ) i , so | β ( k +1)

Original: i − β ′
Revised: i − β ′

Original: ( k +1)
Revised: ( k +1)

Original: i | ≤ | β
Revised: i | ≤ | β

Original: ( k )
Revised: ( k )

Original: i − β ′
Revised: i − β ′

Original: ( k ) i | .
Revised: ( k ) i | .

Original: B. “H ARMLESS T YPE ” N EURONS
Revised: B. “H ARMLESS T YPE ” N EURONS

Original: Section 4 shows that there are two cases of “harmless type”:
Revised: Section 4 shows that there are two cases of “harmless type”:

Original: H 1 and H 2 .
Revised: H 1 and H 2 .

Original: For neurons in H 1 , the derivatives of parameters are estimated in (15), (16) and (17) (same as in G 2 , 2 ).
Revised: For neurons in H 1 , the derivatives of parameters are estimated in (15), (16) and (17) (same as in G 2 , 2 ).

Original: We can have similar analysis as in G 2 , 2 and show that the inequality α i > 0 > α ′ i , β i > β ′ i can be preserved.
Revised: We can have similar analysis as in G 2 , 2 and show that the inequality α i > 0 > α ′ i , β i > β ′ i can be preserved.

Original: Moreover (cid:12)(cid:12)
Revised: Moreover (cid:12)(cid:12)

Original: α i α ′ i (cid:12)(cid:12) increases.
Revised: α i α ′ i (cid:12)(cid:12) increases.

Original: So the neurons either stay in H 1 or become “good type” if (cid:12)(cid:12)
Revised: So the neurons either stay in H 1 or become “good type” if (cid:12)(cid:12)

Original: α i α ′ i (cid:12)(cid:12) > 1 .
Revised: α i α ′ i (cid:12)(cid:12) > 1 .

Original: In particular, neurons in H 1 do no harm to the performance of the model.
Revised: In particular, neurons in H 1 do no harm to the performance of the model.

Original: For neurons in H 2 , 1 [ α i t y 

Revised: For neurons in H 2 , 1 [ α i t y

Original: + α ′ i
Revised: + α ′ i

Original: t y 1 ≥ 0]
Revised: t y 1 ≥ 0]

Original: = 0 , so the derivatives are all equal to 0.
Revised: = 0 , so the derivatives are all equal to 0.

Original: Therefore they are never updated.
Revised: Therefore they are never updated.

Original: Meanwhile they don’t affect the performance of the model since ϕ ( α i t y 

Revised: Meanwhile they don’t affect the performance of the model since ϕ ( α i t y

Original: + α ′ i
Revised: + α ′ i

Original: t y 1 ) =
Revised: t y 1 ) =

Original: and ∆ i = 0 .
Revised: and ∆ i = 0 .

Original: C L EARNING G UARANTEE
Revised: C L EARNING G UARANTEE

Original: In this section, we prove Theorem 6.1, 6.2 and Lemma 6.3.
Revised: In this section, we prove Theorem 6.1, 6.2 and Lemma 6.3.

Original: Proof of Theorem 6.1.
Revised: Proof of Theorem 6.1.

Original: We prove by contradiction.
Revised: We prove by contradiction.

Original: Suppose
Revised: Suppose

Original: [Equation]
Revised: [Equation]

Original: ) then
Revised: ) then

Original: [Equation]
Revised: [Equation]

Original: Furthermore, we claim that µ 0 < δ .
Revised: Furthermore, we claim that µ 0 < δ .

Original: In fact, if µ 0 ≥ δ , since P [ | ∆ − µ 0 | ≤ δ | ℓ ( x ) = 0] ≥ 1 − ϵ by
Revised: In fact, if µ 0 ≥ δ , since P [ | ∆ − µ 0 | ≤ δ | ℓ ( x ) = 0] ≥ 1 − ϵ by

Original: Corollary 4.2, and ∆ ≥ µ 0 − δ ≥ 0 , we have
Revised: Corollary 4.2, and ∆ ≥ µ 0 − δ ≥ 0 , we have

Original: P [∆ ≥ 0 | ℓ ( x ) = 0] ≥ P [ | ∆ − µ 0 | ≤ δ | ℓ ( x ) = 0] ≥ 1 − ϵ, i.e. P [∆ < 0 | ℓ ( x ) = 0] ≤ ϵ , which contradicts (27).
Revised: P [∆ ≥ 0 | ℓ ( x ) = 0] ≥ P [ | ∆ − µ 0 | ≤ δ | ℓ ( x ) = 0] ≥ 1 − ϵ, i.e. P [∆ < 0 | ℓ ( x ) = 0] ≤ ϵ , which contradicts (27).

Original: Let c := µ 0 − µ 1 , then µ 1 = µ 0 − c < δ − c .
Revised: Let c := µ 0 − µ 1 , then µ 1 = µ 0 − c < δ − c .

Original: Again, by Corollary 4.2, for ℓ ( x ) = 1 , ∆ < µ 1 + δ with probability ≥ 1 − ϵ , we have Z = σ (∆) <
Revised: Again, by Corollary 4.2, for ℓ ( x ) = 1 , ∆ < µ 1 + δ with probability ≥ 1 − ϵ , we have Z = σ (∆) <

Original: σ ( µ 1 + δ ) <
Revised: σ ( µ 1 + δ ) <

Original: σ ( − c + 2 δ ) .
Revised: σ ( − c + 2 δ ) .

Original: Then
Revised: Then

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: ] +
Revised: ] +

Original: [Equation]
Revised: [Equation]

Original: 1] <
Revised: 1] <

Original: [Equation]
Revised: [Equation]

Original: 1 + 1 · ϵ < σ (cid:0) − c 2 (cid:1) + ϵ.
Revised: 1 + 1 · ϵ < σ (cid:0) − c 2 (cid:1) + ϵ.

Original: The last step is due to δ =
Revised: The last step is due to δ =

Original: o ( c ) .
Revised: o ( c ) .

Original: Since σ ( − c 2 ) <
Revised: Since σ ( − c 2 ) <

Original: ϵ 2 , E [ Z | ℓ ( x )
Revised: ϵ 2 , E [ Z | ℓ ( x )

Original: = 1] < 3 ϵ 2 .
Revised: = 1] < 3 ϵ 2 .

Original: Combine with (28),
Revised: Combine with (28),

Original: [Equation]
Revised: [Equation]

Original: On the other hand, (cid:12)(cid:12) E
Revised: On the other hand, (cid:12)(cid:12) E

Original: [ ∂L∂b
Revised: [ ∂L∂b

Original: [Equation]
Revised: [Equation]

Original: ] (cid:12)(cid:12) <
Revised: ] (cid:12)(cid:12) <

Original: ϵ 4 implies
Revised: ϵ 4 implies

Original: [Equation]
Revised: [Equation]

Original: which is a contradiction.
Revised: which is a contradiction.

Original: So P [∆ < 0 | ℓ ( x ) = 0] < 4 ϵ .
Revised: So P [∆ < 0 | ℓ ( x ) = 0] < 4 ϵ .

Original: Similarly, P [∆ > 0 | ℓ ( x ) = 1] < 4 ϵ .
Revised: Similarly, P [∆ > 0 | ℓ ( x ) = 1] < 4 ϵ .

Original: Proof of Theorem 6.2.
Revised: Proof of Theorem 6.2.

Original: If the i -th neuron is of “good type”, from Corollary A.4, we find a uniform lower bound of m i 0 − m i 1 in “good type” regimes.
Revised: If the i -th neuron is of “good type”, from Corollary A.4, we find a uniform lower bound of m i 0 − m i 1 in “good type” regimes.

Original: We have min { p − q 2 , Λ 3 } = p − q 2 .
Revised: We have min { p − q 2 , Λ 3 } = p − q 2 .

Original: Next we estimate α i − α
Revised: Next we estimate α i − α

Original: i and β i − β ′
Revised: i and β i − β ′

Original: i .
Revised: i .

Original: Let A ( k ) i :=
Revised: Let A ( k ) i :=

Original: α ( k ) i − α ′
Revised: α ( k ) i − α ′

Original: ( k ) i , B ( k ) i :
Revised: ( k ) i , B ( k ) i :

Original: = β ( k )
Revised: = β ( k )

Original: i − β ′
Revised: i − β ′

Original: ( k ) i .
Revised: ( k ) i .

Original: We have
Revised: We have

Original: A ( k )
Revised: A ( k )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ( k − 1) i − ∂L ∂β ′ ( k − 1)
Revised: ( k − 1) i − ∂L ∂β ′ ( k − 1)

Original: i (cid:19) ,
Revised: i (cid:19) ,

Original: By Theorem B.7, in G 1 and G 2 , 1 , with probability ≥ 1 − O ( 1 n ) , ∂L ∂α i − ∂L
Revised: By Theorem B.7, in G 1 and G 2 , 1 , with probability ≥ 1 − O ( 1 n ) , ∂L ∂α i − ∂L

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: [ Z ] (cid:18)(cid:18) p
Revised: [ Z ] (cid:18)(cid:18) p

Original: [Equation]
Revised: [Equation]

Original: q p
Revised: q p

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: λ − O (log
Revised: λ − O (log

Original: [Equation]
Revised: [Equation]

Original: n ))
Revised: n ))

Original: (cid:19) ≤ − ( β i − β
Revised: (cid:19) ≤ − ( β i − β

Original: [Equation]
Revised: [Equation]

Original: i ) E
Revised: i ) E

Original: [ Z ]
Revised: [ Z ]

Original: [Equation]
Revised: [Equation]

Original: 2 (cid:18) p
Revised: 2 (cid:18) p

Original: [Equation]
Revised: [Equation]

Original: q p + q (cid:19) ∂L ∂β i − ∂L ∂β ′ i ≤ − (
Revised: q p + q (cid:19) ∂L ∂β i − ∂L ∂β ′ i ≤ − (

Original: [Equation]
Revised: [Equation]

Original: i − α ′ i )
Revised: i − α ′ i )

Original: E
Revised: E

Original: [ Z ] (cid:18)(cid:18) p − q p + q (cid:19)λ − O
Revised: [ Z ] (cid:18)(cid:18) p − q p + q (cid:19)λ − O

Original: (log − 12 n ) (cid:19) ≤ − ( α i − α ′ i )
Revised: (log − 12 n ) (cid:19) ≤ − ( α i − α ′ i )

Original: E
Revised: E

Original: [ Z ] λ 2 (cid:18) p − q p + q (cid:19)
Revised: [ Z ] λ 2 (cid:18) p − q p + q (cid:19)

Original: so
Revised: so

Original: A ( k ) i
Revised: A ( k ) i

Original: =
Revised: =

Original: A ( k − 1) i − η k (cid:18) ∂L ∂α ( k − 1) i − ∂L
Revised: A ( k − 1) i − η k (cid:18) ∂L ∂α ( k − 1) i − ∂L

Original: ∂α ′
Revised: ∂α ′

Original: ( k − 1)
Revised: ( k − 1)

Original: i (cid:19) ≥
Revised: i (cid:19) ≥

Original: A ( k − 1)
Revised: A ( k − 1)

Original: i + η k E
Revised: i + η k E

Original: [ Z ( k ) ] λ 2 (cid:18) p − q p + q (cid:19) 2 B ( k − 1) i
Revised: [ Z ( k ) ] λ 2 (cid:18) p − q p + q (cid:19) 2 B ( k − 1) i

Original: =
Revised: =

Original: A ( k − 1) i + λ (cid:18) p − q p + q (cid:19)
Revised: A ( k − 1) i + λ (cid:18) p − q p + q (cid:19)

Original: B ( k − 1)
Revised: B ( k − 1)

Original: i
Revised: i

Original: B ( k ) i =
Revised: B ( k ) i =

Original: B ( k − 1) i − η k (cid:18) ∂L ∂β ( k − 1)
Revised: B ( k − 1) i − η k (cid:18) ∂L ∂β ( k − 1)

Original: i − ∂L ∂β ′ ( k − 1)
Revised: i − ∂L ∂β ′ ( k − 1)

Original: i (cid:19) ≥ B ( k − 1) i + η k E
Revised: i (cid:19) ≥ B ( k − 1) i + η k E

Original: [ Z ( k ) ] λ 2 (cid:18) p − q p + q (cid:19)
Revised: [ Z ( k ) ] λ 2 (cid:18) p − q p + q (cid:19)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: In matrix form:
Revised: In matrix form:

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:33) (29)
Revised: (cid:33) (29)

Original: Similarly, in G 2 , 2 : (cid:32)
Revised: Similarly, in G 2 , 2 : (cid:32)

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:33) (cid:32)
Revised: (cid:33) (cid:32)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: B ( k −
Revised: B ( k −

Original: [Equation]
Revised: [Equation]

Original: (cid:33) (30) in G 2 , 3 :
Revised: (cid:33) (30) in G 2 , 3 :

Original: (cid:32) A ( k )
Revised: (cid:32) A ( k )

Original: i
Revised: i

Original: B (
Revised: B (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:33) ⪰ (cid:32)
Revised: (cid:33) ⪰ (cid:32)

Original: [Equation]
Revised: [Equation]

Original: (cid:0) p − q p + q (cid:1)
Revised: (cid:0) p − q p + q (cid:1)

Original: [Equation]
Revised: [Equation]

Original: (cid:33) (cid:32)
Revised: (cid:33) (cid:32)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: B (
Revised: B (

Original: [Equation]
Revised: [Equation]

Original: (cid:33) (31) where Λ 2 =
Revised: (cid:33) (31) where Λ 2 =

Original: [Equation]
Revised: [Equation]

Original: (1 − log − 13 n ) p + q .
Revised: (1 − log − 13 n ) p + q .

Original: A uniform relation among (29), (30) and (31) can be given by (30).
Revised: A uniform relation among (29), (30) and (31) can be given by (30).

Original: By eigenvalue decomposition, we have
Revised: By eigenvalue decomposition, we have

Original: [Equation]
Revised: [Equation]

Original: i B ( k ) i ≥ (cid:18) 1 + √ 2 λ (cid:18) p − q p + q (cid:19) 2 (cid:19) 2 k (cid:0) 2 − 18 A (0) i + 2 18 B (0)
Revised: i B ( k ) i ≥ (cid:18) 1 + √ 2 λ (cid:18) p − q p + q (cid:19) 2 (cid:19) 2 k (cid:0) 2 − 18 A (0) i + 2 18 B (0)

Original: i (cid:1)≥ A (0) i B (0) i (cid:18)
Revised: i (cid:1)≥ A (0) i B (0) i (cid:18)

Original: 1 + √ 2 λ (cid:18) p − q p + q (cid:19) 2 (cid:19) 2 k .
Revised: 1 + √ 2 λ (cid:18) p − q p + q (cid:19) 2 (cid:19) 2 k .

Original: Therefore we have a uniform lower bound of m i 

Revised: Therefore we have a uniform lower bound of m i

Original: − m i 1 at the k -th epoch in “good type” regime:
Revised: − m i 1 at the k -th epoch in “good type” regime:

Original: [Equation]
Revised: [Equation]

Original: Next we consider the “bad type” regime.
Revised: Next we consider the “bad type” regime.

Original: By Corollary A.4, we have lower bound of m i 

Revised: By Corollary A.4, we have lower bound of m i

Original: − m i 1 in B 1 ,
Revised: − m i 1 in B 1 ,

Original: B 2 and B 3 respectively.
Revised: B 2 and B 3 respectively.

Original: By Theorem B.9, in B 1 and B 2 , | α i − α
Revised: By Theorem B.9, in B 1 and B 2 , | α i − α

Original: i | and | β i − β ′
Revised: i | and | β i − β ′

Original: i | shrink.
Revised: i | shrink.

Original: Moreover, since Λ 1 > Λ 3 5 , we have a uniform lower bound of m i 

Revised: Moreover, since Λ 1 > Λ 3 5 , we have a uniform lower bound of m i

Original: − m i 1 in B 1 and B 2 :
Revised: − m i 1 in B 1 and B 2 :

Original: [Equation]
Revised: [Equation]

Original: since Λ 1 = (1+log − 13 n ) p 2 − q
Revised: since Λ 1 = (1+log − 13 n ) p 2 − q

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: ≤ p
Revised: ≤ p

Original: [Equation]
Revised: [Equation]

Original: Next we show that | α i − α ′ i | and | β i − β ′ i | can only increase by a limited rate in B 3 .
Revised: Next we show that | α i − α ′ i | and | β i − β ′ i | can only increase by a limited rate in B 3 .

Original: From item 4 of
Revised: From item 4 of

Original: Theorem B.7, we have
Revised: Theorem B.7, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore (note that β i < β ′ i )
Revised: Therefore (note that β i < β ′ i )

Original: A ( k )
Revised: A ( k )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: Since E [ Z ( k ) ] ≥ Ω( ϵ ) 6 , and ˜ ϵ
Revised: Since E [ Z ( k ) ] ≥ Ω( ϵ ) 6 , and ˜ ϵ

Original: = o ( ϵ ) , ϵ 

Revised: = o ( ϵ ) , ϵ

Original: = O ( 1 n ) , so η k O (˜ ϵ ) ≤ η k E [ Z ( k ) ] 1 n = 1 n .
Revised: = O ( 1 n ) , so η k O (˜ ϵ ) ≤ η k E [ Z ( k ) ] 1 n = 1 n .

Original: Suppose
Revised: Suppose

Original: A ( k ) i ≥ O (1) , otherwise, A ( k ) i and | B ( k ) i | increase by an even smaller rate.
Revised: A ( k ) i ≥ O (1) , otherwise, A ( k ) i and | B ( k ) i | increase by an even smaller rate.

Original: So we have
Revised: So we have

Original: [Equation]
Revised: [Equation]

Original: By eigenvalue decomposition, we have
Revised: By eigenvalue decomposition, we have

Original: [Equation]
Revised: [Equation]

Original: We obtained the result for “harmless type” neurons directly from Corollary 4.3.
Revised: We obtained the result for “harmless type” neurons directly from Corollary 4.3.

Original: Proof of Lemma 6.3.
Revised: Proof of Lemma 6.3.

Original: Since all parameters are independent standard normal random variables, we have
Revised: Since all parameters are independent standard normal random variables, we have

Original: [Equation]
Revised: [Equation]

Original: By Chebyshev’s inequality we have
Revised: By Chebyshev’s inequality we have

Original: [Equation]
Revised: [Equation]

Original: For neurons initialized as “good type”, we have
Revised: For neurons initialized as “good type”, we have

Original: [Equation]
Revised: [Equation]

Original: Let ρ denote the probability that a neuron is initialized as “good type”.
Revised: Let ρ denote the probability that a neuron is initialized as “good type”.

Original: By G 1 , G 2 and symmetry, ρ = 2 P [ α > α ′ , α + α ′ > 0 , β > β ′ ] .
Revised: By G 1 , G 2 and symmetry, ρ = 2 P [ α > α ′ , α + α ′ > 0 , β > β ′ ] .

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: we have ρ = 14 .
Revised: we have ρ = 14 .

Original: By Chernoff bound, P [ h g ≥ ρ
Revised: By Chernoff bound, P [ h g ≥ ρ

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: Also by Chebyshev’s inequality,
Revised: Also by Chebyshev’s inequality,

Original: P [ (cid:88)
Revised: P [ (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Set k
Revised: Set k

Original: = 110 π ,
Revised: = 110 π ,

Original: P [
Revised: P [

Original: [Equation]
Revised: [Equation]

Original: So we have
Revised: So we have

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: D E XPERIMENTS ON D YNAMICS OF H IDDEN N EURONS
Revised: D E XPERIMENTS ON D YNAMICS OF H IDDEN N EURONS

Original: This experiment verifies our argument in Sections B.1, B.2, B.3 and Theorem 6.2 about the dynamics of hidden neurons.
Revised: This experiment verifies our argument in Sections B.1, B.2, B.3 and Theorem 6.2 about the dynamics of hidden neurons.

Original: We set h = 5 , λ = 0 .
Revised: We set h = 5 , λ = 0 .

Original: 3 and train the model on graphs sampled from SBM with n = 1000 , a = 1 . 0 , b = 0 .
Revised: 3 and train the model on graphs sampled from SBM with n = 1000 , a = 1 . 0 , b = 0 .

Original: The plot of accuracy and its distribution can be seen in Section 7.
Revised: The plot of accuracy and its distribution can be seen in Section 7.

Original: Here we plot the dynamics of all the 5 hidden neurons in Figure 4, with each row corresponding to one hidden neuron.
Revised: Here we plot the dynamics of all the 5 hidden neurons in Figure 4, with each row corresponding to one hidden neuron.

Original: In each plot, x -axis represents epoch and y -axis represents the value of neurons.
Revised: In each plot, x -axis represents epoch and y -axis represents the value of neurons.

Original: The first column depicts α i and α ′ i , the second column |
Revised: The first column depicts α i and α ′ i , the second column |

Original: α i α ′ i | , the third column | α i − α
Revised: α i α ′ i | , the third column | α i − α

Original: i
Revised: i

Original: | , the fourth column β i , β ′ i and the last column | β i − β ′ i | .
Revised: | , the fourth column β i , β ′ i and the last column | β i − β ′ i | .

Original: As shown in the figure, the first, second and fourth neurons are of “good" type satisfying ( G 2 ).
Revised: As shown in the figure, the first, second and fourth neurons are of “good" type satisfying ( G 2 ).

Original: Throughout training these neurons are preserved as “good" type: they’re order-aligned, | α
Revised: Throughout training these neurons are preserved as “good" type: they’re order-aligned, | α

Original: i α ′ i | is lowered bounded by 1, and both | α i − α ′| , | β i − β ′
Revised: i α ′ i | is lowered bounded by 1, and both | α i − α ′| , | β i − β ′

Original: i | keeps increasing.
Revised: i | keeps increasing.

Original: All of these verify our argument in B.1.
Revised: All of these verify our argument in B.1.

Original: The third neuron is “harmless" satisfying ( H 2 ).
Revised: The third neuron is “harmless" satisfying ( H 2 ).

Original: As shown in B.3, this neuron isn’t updated and doesn’t make contribution to the output.
Revised: As shown in B.3, this neuron isn’t updated and doesn’t make contribution to the output.

Original: The fifth neuron is of “bad" type satisfying ( B 2 ).
Revised: The fifth neuron is of “bad" type satisfying ( B 2 ).

Original: Although | α i − α ′
Revised: Although | α i − α ′

Original: i | and | β i − β ′
Revised: i | and | β i − β ′

Original: i | increase, but by comparing with the first, second and fourth row (“good" neurons), they increase at a much smaller.
Revised: i | increase, but by comparing with the first, second and fourth row (“good" neurons), they increase at a much smaller rate.

Original: This verifies our result in Theorem 6.2.
Revised: This verifies our result in Theorem 6.2.

Original: E T ABLE OF N OTATIONS
Revised: E T ABLE OF N OTATIONS

Original: We list the notations used in this paper for readers’ convenience.
Revised: We list the notations used in this paper for readers’ convenience.

Original: [Table]
Revised: [Table]

Original: Definition
Revised: Definition

Original: [Table]
Revised: [Table]

Original: trainable weights in the first
Revised: trainable weights in the first

Original: [Table]
Revised: [Table]

Original: of GCN
Revised: of GCN

Original: [Table]
Revised: [Table]

Original: trainable weights in the second layer of GCN
Revised: trainable weights in the second layer of GCN

Original: [Table]
Revised: [Table]

Original: bias matrix of GCN with each row of B being [ b 0 , b 1 ]
Revised: bias matrix of GCN with each row of B being [ b 0 , b 1 ]

Original: [Table]
Revised: [Table]

Original: bias in the first component
Revised: bias in the first component

Original: [Table]
Revised: [Table]

Original: bias in the second component
Revised: bias in the second component

Original: [Table]
Revised: [Table]

Original: number
Revised: number

Original: [Table]
Revised: [Table]

Original: features
Revised: features

Original: [Table]
Revised: [Table]

Original: logit in the first component without bias
Revised: logit in the first component without bias

Original: [Table]
Revised: [Table]

Original: logit in the second component without bias
Revised: logit in the second component without bias

Original: [Table]
Revised: [Table]

Original: logit in the first component, g 

Revised: logit in the first component, g

Original: = f 

Revised: = f

Original: + b
Revised: + b

Original: [Table]
Revised: [Table]

Original: logit in the second component, g 1 = f 1 + b
Revised: logit in the second component, g 1 = f 1 + b

Original: [Table]
Revised: [Table]

Original: difference between logit, ∆ = g 0 − g 1 = f 0 − f 1 + b 0 − b
Revised: difference between logit, ∆ = g 0 − g 1 = f 0 − f 1 + b 0 − b