Original: A N ALTERNATIVE APPROACH TO TRAIN NEURAL NETWORKS USING MONOTONE VARIATIONAL INEQUALITY
Revised: A N ALTERNATIVE APPROACH TO TRAIN NEURAL NETWORKS USING MONOTONE VARIATIONAL INEQUALITY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The current paper investigates an alternative approach to neural network training, which is a non-convex optimization problem, through the lens of another convex problem — to solve a monotone variational inequality (MVI) - inspired by a recent work of (Juditsky & Nemirovsky, 2019).
Revised: The current paper investigates an alternative approach to neural network training, which is a non-convex optimization problem, through the lens of another convex problem — to solve a monotone variational inequality (MVI) - inspired by a recent work of (Juditsky & Nemirovsky, 2019).

Original: MVI solutions can be found by computationally efficient procedures, with performance guarantee of ℓ 2 and ℓ ∞ bounds on model recovery and prediction accuracy under the theoretical setting of training a single-layer linear neural network.
Revised: MVI solutions can be found by computationally efficient procedures, with performance guarantee of ℓ 2 and ℓ ∞ bounds on model recovery and prediction accuracy under the theoretical setting of training a single-layer linear neural network.

Original: We study the use of MVI for training multi-layer neural networks by proposing a practical and completely general algorithm called stochastic variational inequality ( SVI ).
Revised: We study the use of MVI for training multi-layer neural networks by proposing a practical and completely general algorithm called stochastic variational inequality ( SVI ).

Original: We demonstrate its applicability in training fully-connected neural networks, graph neural networks (GNN), and convolutional networks (CNN) ( SVI is completely general for training other network architectures).
Revised: We demonstrate its applicability in training fully-connected neural networks, graph neural networks (GNN), and convolutional networks (CNN) ( SVI is completely general for training other network architectures).

Original: We show the competitive or better performance of SVI compared to widely-used stochastic gradient descent methods on both synthetic and real network data prediction tasks regarding various performance metrics, especially in the improved efficiency in the early stage of training.
Revised: We show the competitive or better performance of SVI compared to widely-used stochastic gradient descent methods on both synthetic and real network data prediction tasks regarding various performance metrics, especially in the improved efficiency in the early stage of training.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural network (NN) training (Sutskever et al., 2013; Kingma & Ba, 2015; Ioffe & Szegedy, 2015) is the essential process in the study of deep models.
Revised: Neural network (NN) training (Sutskever et al., 2013; Kingma & Ba, 2015; Ioffe & Szegedy, 2015; Yehudai & Shamir, 2019) is the essential process in the study of deep models.

Original: Optimization guarantee for training loss as well as generalization error have been obtained with over-parametrized networks (Neyshabur et al., 2014; Mei et al., 2018; Arora et al., 2019a;b; Allen-Zhu et al., 2019; Du et al., 2019).
Revised: Optimization guarantee for training loss as well as generalization error have been obtained with over-parametrized networks (Neyshabur et al., 2014; Mei et al., 2018; Arora et al., 2019a;b; Allen-Zhu et al., 2019; Du et al., 2019).

Original: However, due to the inherent non-convexity of loss objectives, theoretical developments are still diffused and lag behind the vast empirical successes.
Revised: However, due to the inherent non-convexity of loss objectives, theoretical developments are still diffused and lag behind the vast empirical successes.

Original: Recently, the seminal work (Juditsky & Nemirovsky, 2019) showed how to estimate parameters of  generalized linear models (GLM) via solving a monotone variational inequality (MVI), a general form of convex optimization with strong performance guarantees, when the original problem is non-convex.
Revised: Recently, the seminal work (Juditsky & Nemirovsky, 2019) presented a somehow surprising result that some non-convex issues can be circumvented in special cases by problem reformulation. In particular, it was shown that one can formulate the parameter estimation problem of the generalized linear models (GLM) as solving an monotone variational inequality (MVI), a general form of convex optimization.

Revised: This differs from minimizing a least-square loss function, which leads to a nonconvex optimization problem, and thus no guarantees can be obtained for global convergence nor model recovery.

Revised: Thus, the formulation through MVI leads to strong performance guarantees and computationally efficient procedures.

Original: In this paper, inspired by (Juditsky & Nemirovsky, 2019), we investigate how MVI can be useful in neural network training; to  the best of our knowledge, the current paper is the first to do so.
Revised: In this paper, inspired by (Juditsky & Nemirovsky, 2019) and the fact that certain GLM (such as logistic regression) can be viewed as the simplest network with only one layer, we consider a new scheme for neural network training based on MVI. This is a drastic departure from the widely used gradient descent algorithm for neural network training — we replace the gradient of a loss function with operators inspired by MVI theory. The benefits include (i) in special cases, we can establish strong training and prediction guarantees in Section 4; (ii) for general cases, through extensive numerical studies on synthetic and real-data in Section 5, we demonstrate the faster convergence to a local solution by our approach relative to gradient descent in a comparable setup. To the best of our knowledge, the current paper is the first to study MVI for training neural networks.

Revised: Our SVI , as a general way of modifying the parameter update scheme in NN training, can be readily applied to various deep architectures.

Original: Our SVI is a general way of modifying the parameter update scheme in NN training and can be readily applied to various deep architectures.
Revised: Our SVI is a general way of modifying the parameter update scheme in NN training and can be readily applied to various deep architectures.

Original: In this work, beyond fully-connected (FC) neural networks, we especially study node classification in graph neural networks (GNN) (Wu et al., 2019; Pilanci & Ergen, 2020) and image classification.
Revised: In this work, beyond fully-connected (FC) neural networks, we especially study node classification in graph neural networks (GNN) (Wu et al., 2019; Pilanci & Ergen, 2020) and image classification.

Original: Our technical contributions include:
Revised: Our technical contributions include:

Original: • Develop a general and practical algorithm called stochastic variational inequality ( SVI ).
Revised: • Develop a general and practical algorithm called stochastic variational inequality ( SVI ).

Original: In particular, the algorithm provides a fundamentally different but easy-to-implement alternative to using gradient of the original loss objective with respect to parameters.
Revised: In particular, the algorithm provides a fundamentally different but easy-to-implement alternative to using gradient of the original loss objective with respect to parameters.

Original: In addition, SVI and gradient-based methods barely differ in computation.
Revised: In addition, SVI and gradient-based methods barely differ in computation.

Original: • Reformulate the single/last-layer neural network training as solving a monotone variational inequality, with guarantees on recovery and prediction accuracy.
Revised: Reformulate the single-layer neural network training as solving a monotone variational inequality, with guarantees on recovery and prediction accuracy.

Revised: For multi-layer networks, similar guarantees are obtained assuming ϵ -approximate output from all except the last layer.

Original: • Compare SVI with widely-used stochastic gradient descent methods to demonstrate that SVI is flexible on various tasks and competitive against SGD or Adam, especially the improved efficiency in the early stage of training.
Revised: • Compare SVI with widely-used stochastic gradient descent methods to demonstrate that SVI is flexible on various tasks and competitive against SGD or Adam, especially the improved efficiency in the early stage of training.

Original: Literature.
Revised: Literature.

Original: MVI has been studied mainly in the optimization problem context (Kinderlehrer &
Revised: MVI has been studied mainly in the optimization problem context (Kinderlehrer &

Original: Stampacchia, 1980; Facchinei & Pang, 2003).
Revised: Stampacchia, 1980; Facchinei & Pang, 2003).

Original: More recently, VI has been used to solve min-max problems in Generative Adversarial Networks (Lin et al., 2018) and reinforcement learning (Kotsalis et al., 2020).
Revised: More recently, VI has been used to solve min-max problems in Generative Adversarial Networks (Lin et al., 2018) and reinforcement learning (Kotsalis et al., 2020).

Original: Our theory and techniques are inspired by (Juditsky & Nemirovsky, 2019), but we offer a thorough investigation of using MVI to train multi-layer NN.
Revised: Our theory and techniques are inspired by (Juditsky & Nemirovsky, 2019), but we offer a thorough investigation of using MVI to train multi-layer NN.

Original: In retrospect, our techniques bear similarity to works on “matching loss“ (Amid et al., 2022) but are fundamentally different as we leverage MVI theory and lead to performance guarantees.
Revised: In retrospect, our techniques bear similarity to works on “matching loss“ (Amid et al., 2022) but are fundamentally different as we leverage MVI theory and lead to performance guarantees.

Original: In addition, we differ from (Pilanci & Ergen, 2020), which views two-layer NNs as convex regularizers: we do not convexify the loss minimization, and our SVI extends beyond two-layer networks.
Revised: In addition, we differ from (Pilanci & Ergen, 2020), which views two-layer NNs as convex regularizers: we do not convexify the loss minimization, and our SVI extends beyond two-layer networks.

Original:  P ROBLEM SETUP
Revised:  P ROBLEM SETUP

Original: We briefly describe notation of general NN and GNN of interest and provide preliminaries of MVI.
Revised: We briefly describe notation of general NN and GNN of interest and provide preliminaries of MVI.

Original:  Notations.
Revised: 2.1 NN Notations.

Original: Assume a generic feature X ∈ R C , where C denotes the feature dimension.
Revised: Assume a generic feature X ∈ R C , where C denotes the feature dimension.

Original: Suppose the conditional expectation E [ Y | X ] of the categorical response vector 1 Y ∈ { 1 , . .
Revised: Suppose the conditional expectation E [ Y | X ] of the categorical response vector 1 Y ∈ { 1 , . .

Original: , K } is modeled by an L -layer neural network f ( X, Θ) :
Revised: , K } is modeled by an L -layer neural network G ( X, Θ) :

Original: [Equation]
Revised: [Equation]

Original: where X L = ϕ
Revised: where X L = ϕ

Original: L − 1 ( g L − 1 ( X L − 1 , Θ L − 1 )) , X 

Revised: L − 1 ( g L − 1 ( X L − 1 , Θ L − 1 )) , X

Original: = X denote the nonlinear feature transformation from the previous layer, Θ = { Θ 1 , . . .
Revised: = X denote the nonlinear feature transformation from the previous layer, Θ = { Θ 1 , . . .

Original: , Θ L } denotes model parameters, and each ϕ l denotes the activation function at layer l .
Revised: , Θ L } denotes model parameters, and each ϕ l denotes the activation function at layer l .

Original: In particular, assume there exists Θ ∗ so that E [ Y | X ] = f
Revised: In particular, assume there exists Θ ∗ so that E [ Y | X ] = f

Original: ( X, Θ ∗ ) .
Revised: ( X, Θ ∗ ) .

Original: Ournotation of GNN models (i.e., graph filtering) is standard.
Revised: Our GNN notations are standard and are described in Appendix B.1.

Original: Suppose we have an undirected and connected graph G =

Original: ( V , E , W ) , where V is a finite set of n vertices, E is a set of edges, and

Original: W ∈ R n × n is a weighted adjacency matrix that encodes node connections.

Original: Let I n denote an identity matrix of size n .

Original: Let D be the degree matrix of W and L g =

Original: I n − D − 1 / 2 WD − 1 / 2 be the normalized graph Laplacian, which has the eigen-decomposition L g =

Original: U Λ U T .

Original: For a graph signal X ∈ R n × C with C input channels, it is then filtered via a function g Θ ( L g ) which acts on L g with channel-mixing parameters Θ ∈ R C × F for F output channels.

Original: Thus, the filtered signal X ′ =

Original: g Θ ( L g ) X . In practice, our MVI framework to be introduced next applies to any form of g l ( X l , Θ l ) in (1) for l = 1 , . . .
Revised: In practice, our MVI framework to be introduced next applies to any form of g l ( X l , Θ l ) in (1) for l = 1 , . . .

Original: , L .
Revised: , L .

Original: If g L ( X L , Θ L ) at the last output layer has a particular linear form, we can also obtained theoretical guarantees under idealized assumptions.
Revised: If g L ( X L , Θ L ) at the last output layer has a particular linear form, we can also obtained theoretical guarantees under idealized assumptions.

Original: 2.2 MVI Preliminaries.
Revised: 2.2 MVI Preliminaries.

Original: Given a parameter set Θ ⊂ R p , we call a continuous mapping (operator)
Revised: Given a parameter set Θ ⊂ R p , we call a continuous mapping (operator)

Original: F : Θ → R p monotone if for all Θ 1 , Θ 2 ∈ Θ [Equation] [Equation] Juditsky & Nemirovsky, 2019).
Revised: F : Θ → R p monotone if for all Θ 1 , Θ 2 ∈ Θ , ⟨ F (Θ 1 ) − F (Θ 2 ) , Θ 1 − Θ 2 ⟩ ≥ 0 (Juditsky & Nemirovsky, 2019).

Original: The operator is called strongly monotone with modulus κ if for all Θ 1 , Θ
Revised: The operator is called strongly monotone with modulus κ if for all Θ 1 , Θ

Original: [Equation]
Revised: [Equation]

Original: For a monotone operator F on Θ , the problem VI [ F, Θ ] is to find an ¯Θ ∈ Θ such that for all Θ ∈ Θ ,
Revised: For a monotone operator F on Θ , the problem VI [ F, Θ ] is to find an ¯Θ ∈ Θ such that for all Θ ∈ Θ ,

Original: [Equation]
Revised: [Equation]

Original: It is known that if Θ is compact, then (3) has at least one solution (Outrata et al., 2013, Theorem 4.1).
Revised: It is known that if Θ is compact, then (3) has at least one solution (Outrata et al., 2013, Theorem 4.1).

Original: In addition, if κ > 0 in (2), then(3) has exactly one solution (Outrata et al., 2013, Theorem 4.4).
Revised: In addition, if κ > 0 in (2), then

Original: Under mild computability assumptions, the solution can be efficiently solved to high accuracy, using various iterative schemes (Juditsky & Nemirovsky, 2019).
Revised: Under mild computability assumptions, the solution can be efficiently solved to high accuracy, using various iterative schemes (Juditsky & Nemirovsky, 2019).

Original:  MVI FOR NEURAL NETWORK TRAINING
Revised:  MVI FOR NEURAL NETWORK TRAINING

Original: 3.1 Single-layer training.
Revised: 1 Single-layer training.

Original: Let us first consider a single-layer neural network (i.e., L = 1 in (1)) with a particular form of pre-activation: g ( X, Θ)
Revised: Let us first consider a single-layer neural network (i.e., L = 1 in (1)) with a particular form of pre-activation: g ( X, Θ)

Original: =
Revised: =

Original: η ( X )Θ for an arbitrary feature transformation η .
Revised: η ( X )Θ for an arbitrary feature transformation η .

Original: Although this form can be overly simplistic, it is typically the last layer of a deep neural network.
Revised: Although this form can be overly simplistic, it is typically the last layer of a deep neural network.

Original: For FC networks, η ( X )
Revised: For FC networks, η ( X )

Original: =
Revised: =

Original: X , i.e., the identity map.
Revised: X , i.e., the identity map.

Original: For GNN models, η ( X ) = (cid:80)
Revised: For GNN models, η ( X ) = (cid:80)

Original: Rr =1 h r
Revised: Rr =1 h r

Original: ( L g ) X as the sum of R fixed graph filters determined by graph Laplacian L g (Kipf & Welling, 2017;
Revised: ( L g ) X as the sum of R fixed graph filters determined by graph Laplacian L g (Kipf & Welling, 2017;

Original: Defferrard et al., 2016; Hamilton et al., 2017).
Revised: Defferrard et al., 2016; Hamilton et al., 2017).

Original: One can also write down the corresponding formulation for convolutional layers in image classification under more complex notation.
Revised: One can also write down the corresponding formulation for convolutional layers in image classification under more complex notation.

Original: Note that this is mathematically equivalent to a GLM (Juditsky & Nemirovsky, 2019).
Revised: Note that this is mathematically equivalent to a GLM (Juditsky & Nemirovsky, 2019).

Original: We construct the monotone operator F as
Revised: We construct the monotone operator F as

Original: [Equation]
Revised: [Equation]

Original:  where η ⊺ ( X ) denotes the transpose of η ( X ) .
Revised: 4) where η ⊺ ( X ) denotes the transpose of η ( X ) .

Original: We will explain a few properties of F in Sec. 4, Lemma 4.1.
Revised: We will explain a few properties of F in Sec. 4, Lemma 4.1.

Original: Denote “ F as the empirical sample version of F .
Revised: Denote “ F as the empirical sample version of F .

Original: Then the typical training based on MVI is the projected gradient descent
Revised: Then the typical training based on MVI is the projected gradient descent

Original: [Equation]
Revised: [Equation]

Original:  where γ > 0 is the step-size and Proj Θ projects back estimates to the feasible parameter domain Θ .
Revised: ) where γ > 0 is the step-size and Proj Θ projects back estimates to the feasible parameter domain Θ .

Original: Note that (5) differs from the vanilla stochastic gradient descent (SGD) where the gradient of a certain loss objective takes the role of the monotone operator “ F .
Revised: Note that (5) differs from the vanilla stochastic gradient descent (SGD) where the gradient of a certain loss objective takes the role of the monotone operator “ F .

Original: In particular, “ F needs not correspond to the gradient of any loss function.
Revised: In particular, “ F needs not correspond to the gradient of any loss function.

Original: Nevertheless, we will show in Section 4.3, Proposition 4.7 that “ F is exactly the gradient of the cross-entropy loss, whereby (5) coincides with SGD.
Revised: Nevertheless, we will show in Section 4.3, Proposition 4.8 that “ F is exactly the gradient of the cross-entropy loss, whereby (5) coincides with SGD.

Original: 3.2 Multi-layer training.
Revised: 3.2 Multi-layer training.

Original: We first remark that even if for all l < L with L > 1 , pre-activation g l ( X l , Θ l ) = η l ( X l )Θ l is linear with respect to Θ l , we cannot verify monotonicity of the operator
Revised: Our goal is to train multi-layer networks based on (4). Specifically, we want to obtain F l (Θ l ) for parameters Θ l in layer l of the neural network in (1).

Original:  F ( ‹ Θ) := [ F 1 (Θ 1 ) , . . , F L (Θ L )] . Fundamentally, this is because the original empirical loss minimization problem is non-convex, but solving MVI is a convex problem without convex relaxation.
Revised: These { F l (Θ l ) } Ll = are then used during optimization (e.g., via projected gradient descent (5) or Adam (Kingma & Ba, 2015)).

Original: Nevertheless, motivated by the MVI problem in single-layer settings, SVI in Algorithm 1 heuristically extends MVI to training any neural network described by (1); Figure 1 illustrates the idea.
Revised: However, there are several fundamental difficulties of extending F (Θ) in (4) to training generic multilayer neural networks.

Original: [Equation] [Figure] There are two important benefits of Algorithm 1. First, it is easy to implement and directly leverages the benefit of automatic differentiation (Paszke et al., 2017).
Revised: First, (4) requires that the pre-activation mapping η ( X )Θ is linear in Θ .

Original: In short, it “skips” computing derivatives of the point-wise nonlinearities with respect to inputs.
Revised: This linearity assumption does not hold for many network layer types.

Original: We implement the skipping idea via backpropagating another loss (cid:101) L (Θ l ) (see in line 4) on the parameters.
Revised: Second, more importantly, (4) is defined using Y , the observation of response variable.

Original: This loss (cid:101) L (Θ l ) is simple to compute, as the quantity ‹ X l +1 is available during the forward pass on data and thegradient ∇ X l +1 L is available upon backpropagating the original loss with respect to output of the current layer l . Second, it is generalizable by allowing arbitrary form of g l ( X l , Θ l ) and loss function
Revised: However, such observation is unavailable when we define F l (Θ l ) for intermediate layers, where Y l denotes the response after the l -th hidden layer. Third, note that (4) does not depend on any loss objective (e.g., mean-squared error, cross-entropy loss, etc.), so that the extension under a specific loss objective when training the network is unclear.

Original: L . Lastly, SVI is very flexible as one can apply SVI to all or a subset of layers in the network.
Revised: Despite the difficulties, we propose an MVI-based scheme to train generic neural networks under any loss objective.

Original: We now briefly discuss the motivation of Algorithm 1 based on a mathematically equivalent view of F (Θ) in (4).
Revised: To motivate the scheme, we first present a mathematically equivalent view of F (Θ) in (4).

Original: Assume we have the mean-squared-error (MSE) loss L ( ˆ Y , Y ) :
Revised: Assume we have the mean-squared-error (MSE) loss L ( ˆ Y , Y ) :

Original: = 1 / 2 ∥ ˆ Y − Y ∥ 2 2 and the particular form g ( X, Θ) =
Revised: = 1 / 2 ∥ ˆ Y − Y ∥ 22 and the linear pre-activation mapping ‹ X :=

Original: η ( X )Θ .
Revised: η ( X )Θ .

Original: Denote ˆ Y :=
Revised: Denote ˆ Y :=

Original: ϕ ( η ( X )Θ) .
Revised: ϕ ( ‹ X ) as the prediction.

Original: By chain rule
Revised: By chain rule

Original: [Equation]
Revised: [Equation]

Revised: Note that (6) comprises of two terms.

Revised: The first term ∇ ˆ Y L is the gradient of the loss objective with respect to the network output.

Original: This equivalence in (6) motivates step 4 in Algorithm 1.
Revised: The second term ∇ Θ ‹ X is the gradient of the pre-activation with respect to the parameter. In particular, (6) is well-defined for any loss objective L and at any hidden layer l —we would simply replace ‹ X with ‹ X l , the pre-activation mapping at the l -th layer. Therefore, based on the observation (6), SVI in Algorithm 1 heuristically extends MVI to training any neural network described by (1); Figure 1 illustrates the idea. Notation-wise, the subscript i denotes sample index and the subscript l denotes the output of first l network layers.

Original: In retrospect, comparing with the commonly used gradient ∇ Θ L , Algorithm 1 only differs by skipping the additional computation ∇ η ( X )Θ ˆ Y = ∇ η ( X )Θ ϕ ( η ( X )Θ) .
Revised: In retrospect, comparing with the commonly used gradient ∇ Θ L , Algorithm 1 only differs by skipping the additional computation ∇ η ( X )Θ ˆ Y = ∇ η ( X )Θ ϕ ( η ( X )Θ) .

Original: This operation is precisely the derivative of the point-wise non-linearity ϕ .
Revised: This operation is precisely the derivative of the point-wise non-linearity ϕ .

Original: Therefore, SVI barely differs in terms of computational cost against gradient-based methods.
Revised: Therefore, SVI barely differs in terms of computational cost against gradient-based methods.

Original: In practice, this skipping leads to a different neuron update dynamics—see Remark 3.1 for details.
Revised: In practice, this skipping leads to a different neuron update dynamics—see Remark 3.1 for details.

Revised: [Figure]

Original: Remark 3.1 (Effect on training dynamics) . 
Revised: There are several important benefits of Algorithm 1. First, it addresses the aforementioned challenges of extending MVI to multi-layer training—the Algorithm is applicable to arbitrary form of g l ( X l , Θ l ) and loss function L . It requires no observation of responses from hidden layers. Second, it is easy to implement upon leveraging the benefit of automatic differentiation (Paszke et al., 2017).

Revised: We implement the skipping idea via backpropagating another loss (cid:101) L (Θ l ) as in line 4 on the parameters.

Revised: This loss (cid:101) L (Θ l ) is simple to compute, as the quantity ‹ X

Revised: l +1 is available during the forward pass on data and the gradient ∇ X l +1 L is available upon backpropagating the original loss with respect to output of the current layer l .

Revised: Third, SVI is very flexible as one can apply SVI to all or a subset of layers in the network.

Revised: Remark 3.1 (Effect on training dynamics) .

Original: We remark a key difference between parameter update in SGD and in SVI , which ultimately affects training dynamics.
Revised: We remark a key difference between parameter update in SGD and in SVI , which ultimately affects training dynamics.

Original: Suppose the activation function is
Revised: Suppose the activation function is

Original: ReLU.
Revised: ReLU.

Original: It is well-known that SGD does not update weights of inactive neurons because the gradient of ReLU with respect to them is zero.
Revised: It is well-known that SGD does not update weights of inactive neurons because the gradient of ReLU with respect to them is zero.

Original: However, SVI does not discriminate as it skips this derivative computation.
Revised: However, SVI does not discriminate as it skips this derivative computation.

Original: Thus, one can expect that SVI results in further weight updates than SGD, which experimentally seems to speed up the initial model convergence.
Revised: Thus, one can expect that SVI results in further weight updates than SGD, which experimentally seems to speed up the initial model convergence.

Original: We illustrate this phenomenon in
Revised: We illustrate this phenomenon in

Original: Figure 4 and will provide explanations in future works.
Revised: Figure 4 and will provide explanations in future works.

Revised: Algorithm

Revised: Require: Inputs are (a) Training data { ( X i , Y i ) } Ni =1 (b)

Revised: An L -layer network G ( X, Θ) :

Revised: = { ϕ l ( g l ( X l , Θ l )) }

Original:  G UARANTEE OF MODEL RECOVERY BY MVI
Revised: Ll =1 (c) Boolean mask BM of length L (d) Loss function L := L ( G ( X, Θ) , Y ) Ensure: Estimated parameters ˆΘ := { ˆΘ l } Ll = 1: while Training not stopped do 2: for

Revised: Layer l = L

Revised: , 1 do 3: if is True then {

Revised: Use SVI

Original: We now present guarantees on model recovery for the last-layer training when g L ( X L , Θ L ) =
Revised: Compute F l ( := L over mini-batches, where : end if 9: end for 10: end while

Original:  η L ( X L )Θ L) in (1) and previous layers are fully known; in practice, this can be understood as if previous layers have provided necessary feature extraction.
Revised: We now present training and estimation guarantees on model recovery for the last-layer training when g L ( X L , Θ L ) = η L ( X L )Θ L in (1) and previous layers are estimated with ϵ accuracy.

Original: Let { ( X i , Y i ) } Ni =1 be N training data from model (1) with Θ = Θ ∗ .
Revised: Let { ( X i , Y i ) } Ni =1 be N training data from model (1) with Θ = Θ ∗ .

Original: For the subsequent theoretical analyses, we assume that for each i , the quantity X ∗ i,L is known, which is the output from the previous L − 1 hidden layers using X i and  the true parameters { Θ ∗ 1 , . . . , Θ ∗ L − 1 } .
Revised: Let f ∗ 1: L − 1 ( · ) and ˆ f 1: L − 1 ( · ) be the oracle and estimated mapping from the previous L − 1 layers, and let X ∗ i,L and ˆ X i,L be the corresponding outputs from the mappings. For the subsequent theoretical analyses, we will assume that the expected ℓ p difference of these two mappings are close enough (see Assumption 4.2).

Original: Thus, η ( X )  becomes η (  X∗ L ) for a generic feature X .
Revised: In this setting, η ( X ) thus becomes η ( ˆ X L ) for a generic feature X .

Original: For the operator F (Θ L ) in (4), where Θ is the parameter space of Θ L , consider its empirical average F 1: N (Θ L ) :
Revised: For the operator F (Θ L ) in (4), where Θ is the parameter space of Θ L , consider its empirical average F 1: N (Θ L ) :

Original: = (1 /N )
Revised: = (1 /N )

Original: (cid:80) Ni =
Revised: (cid:80) Ni =

Original: 1 F i (Θ L ) , where F i (Θ L ) :
Revised: 1 F i (Θ L ) , where F i (Θ L ) :

Original: = η ⊺ (  X∗ i,L )[ ϕ ( η (  X∗ i,L )Θ L ) − Y ] .
Revised: = η ⊺ ( ˆ X i,L )[ ϕ ( η ( ˆ X i,L )Θ L ) − Y ] .

Original: Let “ Θ ( T ) L be the estimated parameter after T training iterations, using F 1: N and (5).
Revised: Let “ Θ ( T ) L be the estimated parameter after T training iterations, using F 1: N and (5).

Original: For a test feature X t and its nonlinear transformation  X∗ t,L , consider
Revised: For a test feature Y t and the transformation ˆ X t,L , consider

Original: [Equation]
Revised: [Equation]

Original: which is the estimated prediction using “ Θ ( T ) L and will be measured against the true model
Revised: which is the estimated prediction using “ Θ ( T ) L and will be measured against the true model

Original: E [ Y t | X t ] under Θ ∗ L  .
Revised: E [ Y t | Y t ] under Θ ∗ L and X ∗ t,L .

Original: In particular, we will provide error bound on ∥ (cid:98) E
Revised: In particular, we will provide error bound on ∥ (cid:98) E

Original: [ Y t | X t ] − E
Revised: [ Y t | Y t ] − E

Original: [ Y t | X t ] ∥ p , p ≥ 2 which crucially depends on the strong monotonicity modulus κ in (2) for F (Θ L ) .
Revised: [ Y t | Y t ] ∥ p , p ≥which crucially depends on the strong monotonicity modulus κ in (2) for F (Θ L ) .

Original: Wefirst state a few properties of F (Θ L ) defined in (4), which explicitly identify the form of κ .
Revised: We state a few properties of F (Θ L ) defined in (4), which explicitly identify the form of κ .

Original: All proofs of Lemmas and Theorems, as well as certain remarks, are contained in Appendix A.
Revised: All proofs of Lemmas and Theorems, as well as certain remarks, are contained in Appendix A.

Original: Lemma 
Revised: Lemma 4.1.

Original: Assume ϕ
Revised: Assume ϕ

Original: L is K -Lipschitz continuous and monotone on its domain
Revised: L is K -Lipschitz continuous and monotone on its domain.

Original: For an input X , 1.
Revised: For an input X , 1.

Original: F (Θ L ) is both monotone and K 2 -Lipschitz, where K 2 :=
Revised: F (Θ L ) is both monotone and K 2 -Lipschitz, where K 2 :=

Original: K E X {∥ η
Revised: K E X {∥ η

Original: (  X∗ L ) ∥ 
Revised: ( ˆ X L ) ∥ 22 .

Original: κ = λ min ( ∇ ϕ L ) E X [ λ min ( η ⊺ ( X ∗ L ) η ( X ∗ L ))]
Revised: [Equation]

Original:  if ∇ ϕ L exists; it is 0 otherwise.F (Θ ∗ L ) = 0 .
Revised: ))] if ∇ ϕ L exists; it is 0 otherwise.

Original: For simplicity, we assume from now on the stronger condition that λ min ( η ⊺ (  X∗ L ) η
Revised: For simplicity, we assume from now on the stronger condition that λ min ( η ⊺ ( ˆ X L ) η

Original: (  X∗ L )) > 0 for any generic nonlinear feature  X∗ L .
Revised: ( ˆ X L )) > 0 for any generic nonlinear feature ˆ X L .

Original: In addition, because we are considering the last layer in classification, ϕ L is typically chosen as sigmoid or softmax, both of which are differentiable so λ min ( ∇ ϕ L ) exits.
Revised: In addition, because we are considering the last layer in classification, ϕ L is typically chosen as sigmoid or softmax, both of which are differentiable so λ min ( ∇ ϕ L ) exits.

Original: 4. C ASE 1: MODULUS κ >
Revised: 4. C ASE 1: MODULUS κ >

Original: Note the property F (Θ ∗  L  ) [-= 
-]
Revised: We first state several assumptions used in convergence analyses. Assumption 4.2 ( ϵ -approximate estimate from previous layers) . Let ˆ X L := f ∗ 1: L − 1 ( X ) [-= 
-]and ˆ X L :=

Original: implies that Θ ∗ L is a solution to VI [ F, Θ ] .
Revised: ˆ f 1: L − 1 ( X ) be the oracle and estimated output from the previous L − 1 layers.

Original: In particular, if κ > 0 , it is the unique solution.
Revised: There exists ϵ >such that E X ∥ ˆ X L − ˆ X L ∥ 2 ≤ ϵ. Assumption 4.

Original: As a result, Θ ∗ L can be efficiently solved to a high accuracy using F (Θ L ) under appropriated chosen γ in (5).
Revised: (1) The oracle parameter Θ ∗ L of the last layer satisfies the MVI inequality (3) for F . The mapping η ( · ) is D -Lipschitz continuous. For any Θ L ∈ Θ , ∥ Θ L ∥ 2 ≤ B .

Original: Thus, we bound the recovered parameters following techniques in (Juditsky & Nemirovsky, 2019).
Revised: We bound the recovered parameters following techniques in (Juditsky & Nemirovsky, 2019).

Original: Lemma 4.2 (Parameter recovery guarantee) .
Revised: Lemma 4.4 (Parameter recovery guarantee) .

Original: Suppose that there exists M < ∞ such that ∀ Θ L ∈ Θ ,
Revised: Suppose that there exists M < ∞ such that ∀ Θ L ∈ Θ ,

Original: [Equation]
Revised: [Equation]

Original: where E [ Y Θ L | X ] = ϕ
Revised: where E [ Y Θ L | X ] = ϕ

Original: ( η (  X∗ L )Θ L ) .
Revised: ( η ( ˆ X L )Θ L ) .

Original: Choose adaptive step sizes γ = γ t :=
Revised: Choose adaptive step sizes γ = γ t :=

Original: [ κ ( t + 1)] − 1 in (5) .
Revised: [ κ ( t + 1)] − 1 in (5) .

Original: The sequence of estimates “ Θ ( T ) L obeys the error bound
Revised: The sequence of estimates “ Θ ( T ) L obeys the error bound

Original: [Equation]
Revised: [Equation]

Original: We can use the above result to bound the error in posterior prediction.
Revised: We can use the above result to bound the error in posterior prediction.

Original: Theorem 4.3 (Prediction error bound for model recovery, strongly monotone F ) .
Revised: Theorem 4.5 (Prediction error bound for model recovery, strongly monotone F ) .

Original: We have for a given test signal X t , t > N that for p ∈ [2 , ∞ ] ,
Revised: Under Assumptions 4.2 and 4.3, we have for a given test signal X t , t > N that for p ∈ [2 , ∞ ] ,

Original: [Equation]
Revised: [Equation]

Original: where  C t :=
Revised: where (cid:98) E [ Y t | X t ] is defined in (7) under constants C t :=

Original: [4 M 2 Kλ max ( η ⊺ (  X∗ t,L ) η (  X∗ t,L ))] 
Revised: [4 M 2 Kλ max ( η ⊺ ( ˆ X t,L ) η ( ˆ X t,L ))] /κ 2 and

Original: /κ 2 and (cid:98) E [ Y t | X t ] is defined in (7) .
Revised: C:=KDB.

Original: In particular, p = 2 yields the sum of squared error bound on prediction and p = ∞ yields entry-wise bound.
Revised: In particular, p = 2 yields the sum of squared error bound on prediction and p = ∞ yields entry-wise bound.

Original: Note that the guarantee is point-wise, as we can bound the parameter recovery error in expectation in Lemma 4.2.

Original: Moreover, the same order of convergence holds when F in (4) is estimated by the empirical average of mini-batches of training data.
Revised: The same order of convergence holds when F in (4) is estimated by the empirical average of minibatches of training data.

Original: In particular, the proof of Theorem 4.3 only requires access to an unbiased estimator of F so that the batch size can range from one to N , where N is the size ofthe training data.Corollary 4.4 (Prediction error bound for test loss) . Fix an ϵ > 0 .
Revised: In particular, the proof of Theorem 4.5 only requires access to an unbiased estimator of F so that the batch size can range from one to N , where N is the size of training data.

Original: For any test signal X t , assume

Original: N > C t /ϵ .

Original: Denote L ∗ ( X t , Y t ) (resp.

Original: ˆ L ( X t , Y t ) ) as the binary cross-entropy loss under the true (resp.

Original: estimated) model.

Original: Then,

Original: [Equation]

Original: where E t,i :=

Original: E [ Y t | X t ] i is the true conditional expectation of Y t at node i , 1 and ϵ are vectors ofand ϵ , and the division is point-wise.

Original: Remark 4.5 (When κ > 0 ) .
Revised: Remark 4.6 (When κ > 0 ) .

Original: Recall that κ = λ min ( ∇ ϕ L )
Revised: Recall that κ = λ min ( ∇ ϕ L )

Original: E X
Revised: E X

Original: [ λ min ( η ⊺ (  X∗ L ) η
Revised: [ λ min ( η ⊺ ( ˆ X L ) η

Original: (  X∗ L ))] .
Revised: ( ˆ X L ))] .

Original: We may assume that the quantity E X [ λ min ( η ⊺ (  X∗ L ) η (  X∗ L ))] is always lower bounded away from zero, so as to only concern with λ min ( ∇ ϕ L ) .
Revised: We may assume that the quantity E X [ λ min ( η ⊺ ( ˆ X L ) η ( ˆ X L ))] is always lower bounded away from zero, so as to only concern with λ min ( ∇ ϕ L ) .

Original: When ϕ L is a point-wise function on its vector inputs, this gradient matrix is diagonal.
Revised: When ϕ L is a point-wise function on its vector inputs, this gradient matrix is diagonal.

Original: In the case of the sigmoid function, we know that for any y ∈ R n
Revised: In the case of the sigmoid function, we know that for any y ∈ R n

Original: [Equation]
Revised: [Equation]

Original: which is bounded away from zero.
Revised: which is bounded away from zero.

Original: In general, we only need that the point-wise activation is continuously differentiable with positive derivatives.
Revised: In general, we only need that the point-wise activation is continuously differentiable with positive derivatives.

Original: 4. C ASE 2: MODULUS κ =
Revised: 4. C ASE 2: MODULUS κ =

Original: We may also encounter cases where the operator F is only monotone but not strongly monotone.
Revised: We may also encounter cases where the operator F is only monotone but not strongly monotone.

Original: For instance, let ϕ be the softmax function, which satisfies ∇ ϕ ( z ) 1 = 0 for any z ∈ R n (Gao & Pavel, 2018, Proposition 2).
Revised: For instance, let ϕ be the softmax function, which satisfies ∇ ϕ ( z ) 1 = 0 for any z ∈ R n (Gao & Pavel, 2018, Proposition 2).

Original: Then, the minimum eigenvalue of ∇ ϕ ( z ) is always zero, leading to κ = 0 .
Revised: Then, the minimum eigenvalue of ∇ ϕ ( z ) is always zero, leading to κ = 0 .

Original: In this case, the solution of VI [ F, Θ ] in (3) needs not be unique.

Original: Nevertheless, wecan use the extrapolation methoed (OE) (Kotsalis et al., 2020) toapproximate the zero of F (¯Θ L ) = E X,Y { η ( X ∗ L ) [ ϕ L ( η ⊺ ( X ∗ L ))¯Θ L ] − Y } and obtain  similar but weaker ℓ p prediction guarantee.
Revised: In this case, we use the extrapolation methoed (OE) (Kotsalis et al., 2020) to obtain a similar but weaker ℓ p prediction guarantee.

Original: Theorem 4.6 (Prediction error bound for model recovery, monotone F ) .
Revised: Theorem 4.7 (Prediction error bound for model recovery, monotone F ) .

Original: Suppose we run the OE algorithm (Kotsalis et al., 2020) for T iterations with λ t = 1 , γ t =
Revised: Suppose we run the OE algorithm (Kotsalis et al., 2020) for T iterations with λ t = 1 , γ t =

Original: [4 K 2 ] − 1 , where K 2 is
Revised: [4 K 2 ] − 1 , where K 2 is

Original: the
Revised: the

Original: Lipschitz constant of F .
Revised: Lipschitz constant of F .

Original: Let R be uniformly chosen from { 2 , 3 , . . .
Revised: Let R be uniformly chosen from { 2 , 3 , . . .

Original: , T } .
Revised: , T } .

Original: Then for p ∈ [2 , ∞ ] ,
Revised: Then for p ∈ [2 , ∞ ] ,

Original: [Equation]
Revised: [Equation]

Original: where σ min ( · ) denotes the minimum singular value of its input matrix and the constant C ′′ t :
Revised: where σ min ( · ) denotes the minimum singular value of its input matrix and the constant C ′′ t :

Original: = 3 σ + 12 K 2 » 2 ∥
Revised: = 3 σ + 12 K 2 » 2 ∥

Original: Θ ∗ L ∥
Revised: Θ ∗ L ∥

Original: + 2 σ 2 /L 2 , in which σ 2 :=
Revised: + 2 σ 2 /L 2 , in which σ 2 :=

Original: E [( F i (Θ L ) − F (Θ L )) 2 ] is the variance of the unbiased estimator.
Revised: E [( F i (Θ L ) − F (Θ L )) 2 ] is the variance of the unbiased estimator.

Original: The convergence rate in Theorem 4.6 is also unaffected by the batch size, which only serves to reduce the variance.
Revised: The convergence rate in Theorem 4.7 is also unaffected by the batch size, which only serves to reduce the variance.

Original: In addition, Theorem 4.6 requires R be uniformly chosen from { 2 , 3 , . .
Revised: In addition, Theorem 4.7 requires R be uniformly chosen from { 2 , 3 , . .

Original: , T } , so that the theoretical guarantee holds at a random training epoch.
Revised: , T } , so that the theoretical guarantee holds at a random training epoch.

Original: In theory, this assumption is necessary to ensure a decrease of the norm of the monotone operator ((Kotsalis et al., 2020), Eq.
Revised: In theory, this assumption is necessary to ensure a decrease of the norm of the monotone operator ((Kotsalis et al., 2020), Eq.

Original: In practice, we observed that the epoch that leads to the highest validation accuracy might not occur at the end of T training epochs, so this assumption is reasonable based on empirical evidence.
Revised: In practice, we observed that the epoch that leads to the highest validation accuracy might not occur at the end of T training epochs, so this assumption is reasonable based on empirical evidence.

Original: 4. E QUIVALENCE BETWEEN MONOTONE OPERATOR AND GRADIENT OF PARAMETERS
Revised: 4. E QUIVALENCE BETWEEN MONOTONE OPERATOR AND GRADIENT OF PARAMETERS

Original: In practice, neural networks, including single-layer ones, are commonly trained via empirical loss minimization, in contrast to solving MVI.
Revised: In practice, neural networks, including single-layer ones, are commonly trained via empirical loss minimization, in contrast to solving MVI.

Original: We now show that under the cross-entropy loss, when ϕ is either the sigmoid function or the softmax function, the monotone operator F defined in (4) coincides with the gradient of the loss with respect to parameters.
Revised: We now show that under the cross-entropy loss, when ϕ is either the sigmoid function or the softmax function, the monotone operator F defined in (4) coincides with the gradient of the loss with respect to parameters.

Original: Proposition 4.7 (The equivalence between MVI and parameter gradient) .
Revised: Proposition 4.8 (The equivalence between MVI and parameter gradient) .

Original: Consider a generic pair of input signal X and random realization Y in binary or multi-class classification with the sigmoid or softmax function under cross-entropy loss.
Revised: Consider a generic pair of input signal X and random realization Y in binary or multi-class classification with the sigmoid or softmax function under cross-entropy loss.

Original: For any parameter Θ
Revised: For any parameter Θ

Original: [Equation]
Revised: [Equation]

Original: where the monotone operator F (Θ L ) substitutes η L ( X L ) and Θ L in (4) .
Revised: where the monotone operator F (Θ L ) substitutes η L ( X L ) and Θ L in (4) .

Original: 4. I MPRECISE GRAPH KNOWLEDGE
Revised: 4. I MPRECISE GRAPH KNOWLEDGE

Original: Because graph knowledge is rarely perfectly known, the robustness of the algorithm under estimated graph topology knowledge is important in practice.
Revised: Because graph knowledge is rarely perfectly known, the robustness of the algorithm under estimated graph topology knowledge is important in practice.

Original: Thus, we want to analyze the difference in prediction quality when the true graph adajacency matrix W is estimated by another W ′ .
Revised: Thus, we want to analyze the difference in prediction quality when the true graph adajacency matrix W is estimated by another W ′ .

Original: For simplicity, we focus on the GCN case and denote η (  X∗ L ) under true or estimated graph Laplacian as L g X or L ′ g X .
Revised: For simplicity, we focus on the GCN case and denote η ( ˆ X L ) under true or estimated graph Laplacian as L g X or L ′ g X .

Original: For a given k ∈ { 1 , . . . , n } , decompose the graph Laplacian L g as L g =

Original: L + g + L − g , where L + g := U Λ + U T , Λ + = diag ( λ 1 , . . . , λ k , 0 ) denotes the high-energy portion of the filter (similarly for L − g ). We show that when the high/low energy portions are similar, there exists a solution Θ ∗ L ′ bounding the prediction error between E [ Y | X ] under Θ ∗ L and Θ ∗ L ′ .
Revised: We show that when L g and L ′ g are close in spectral norm, there exists a solution Θ ∗ L ′ bounding the prediction error between E [ Y | X ] under Θ ∗ L and Θ ∗ L ′ .

Original: In particular, Θ ∗ L ′ is the solution of VI [ F, Θ ] if ϕ L is the sigmoid function so that we can guarantee a small prediction error in expectation based on earlier results in such cases.
Revised: In particular, Θ ∗ L ′ is the solution of VI [ F, Θ ] if ϕ L is the sigmoid function so that we can guarantee a small prediction error in expectation based on earlier results in such cases.

Original: We also experimentally verify the performance of SVI under model mismatch in Section 5.1.2.
Revised: We also experimentally verify the performance of SVI under model mismatch in Section 5.1.

Original: Proposition 4.8 (Quality of minimizer under model mismatch) .
Revised: Proposition 4.9 (Quality of minimizer under model mismatch) .

Original: Fix an ϵ > 0 .
Revised: Fix a δ > 0 .

Original: For a given k ∈ { 1 , . . . , n } , decompose L g (or L ′ g) as L g = L + g + L −g , where L + g := U Λ + U T , Λ + = diag ( λ 1 , . . .
Revised: Assume that ∥ L ′ g −

Original: , λ k , 0 ) , λ 1 ≤ λ 2 . . . Assume that (1) L + g = L′ + g (2) ∥ L ′ − g − L − g ∥ 2 ≤ ϵ/ [ K E X 
Revised: L g ∥ 2 ≤ δ/ [ K E X ∥

Original: ∥ X− ∥  ] , where K is the Lipschitz continuous constant for ϕ Land X − belongs to the span of L − g and L ′ − g .
Revised: X ∥ 2 ] , where K is the Lipschitz continuous constant for ϕ L .

Original: Then, there exists Θ ∗ L ′ such that
Revised: Then, there exists Θ ∗ L ′ such that

Original: [Equation]
Revised: [Equation]

Original: where E [ Y | X ] ′ denotes the conditional expectation under L ′ g and Θ ∗ L ′ .
Revised: where E [ Y | X ] ′ denotes the conditional expectation under L ′ g and Θ ∗ L ′ .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We test and compare SVI in Algorithm 1 with widely-used stochastic gradient descent methods on various synthetic and real-data examples.
Revised: We test and compare SVI in Algorithm 1 with widely-used stochastic gradient descent methods on various synthetic and real-data examples.

Original: In particular, we compare SVI or SVI -Adam with SGD or Adam, where SVI (resp.
Revised: In particular, we compare SVI or SVI -Adam with SGD or Adam, where SVI (resp.

Original: SVI -Adam) uses SVI in Algorithm 1 to provide update direction and optimized with SGD (resp.
Revised: SVI -Adam) uses SVI in Algorithm 1 to provide update direction and optimized with SGD (resp.

Original: Adam).
Revised: Adam).

Original: We demonstrate the competitive or better performance by SVI , especially in the improved efficiency in the early stage of training.
Revised: We demonstrate the competitive or better performance by SVI , especially in the improved efficiency in the early stage of training against the SGD baseline.

Original: For fair experimental comparisons, all hyperparameters for SVI / SVI -Adam and SGD/Adam are tuned to be the same.
Revised: For fair experimental comparisons, all hyperparameters for SVI / SVI -Adam and SGD/Adam are tuned to be the same.

Original: Doing so is possible as SVI only provides alternative update directions during the network training,
Revised: Doing so is possible as SVI only provides alternative update directions during the network training,

Original: whereby the same optimization algorithm
Revised: whereby the same optimization algorithm

Original: (e.g., SGD or Adam) can be subsequently used.
Revised: (e.g., SGD or Adam) can be subsequently used.

Original: Partial results are shown due to space limitation.
Revised: Partial results are shown due to space limitation.

Original: Appendix B.1 describes details regarding experiment setup and comparison metrics and B.2 onward contains full results.
Revised: Appendix B.2 describes details regarding experiment setup and comparison metrics and B.3 onward contains full results.

Original: Notation-wise, N (resp.
Revised: Notation-wise, N (resp.

Original: N 1 ) denotes the size of training (resp.
Revised: N 1 ) denotes the size of training (resp.

Original: test) sample, lr denotes the learning rate, B denotes the batch size, and E denotes training epochs.
Revised: test) sample, lr denotes the learning rate, B denotes the batch size, and E denotes training epochs.

Original: 5. S YNTHETIC D ATA E XPERIMENTS5.1.1 Two-layer FC
Revised: 5. S YNTHETIC D ATA E XPERIMENTS

Original:  network classification.
Revised: 5.1.1 Two-layer FC network classification.

Original: We first classify the cluster label of the simulated twomoon dataset.
Revised: We first classify the cluster label of the simulated twomoon dataset.

Original: Figure 9a visualizes the dataset.
Revised: Figure 8a visualizes the dataset.

Original: We use a two-layer fully-connected network with
Revised: We use a two-layer fully-connected network with

Original: ReLU (layer 1) and softmax (layer 2) activation.
Revised: ReLU (layer 1) and softmax (layer 2) activation.

Original: We let N = N 1 = 500 , B = 100 , E = 100 , lr =0.15, and momentum = 0.99.
Revised: We let N = N 1 = 500 , B = 100 , E = 100 , lr =0.15, and momentum = 0.99.

Original: Table 1 shows that SVI consistently reaches smaller MSE losses and classification errors on training and test data across different number of hidden neurons.
Revised: Table 1 shows that SVI consistently reaches smaller MSE losses and classification errors on training and test data across different number of hidden neurons.

Original: Figureshows that SVI also converges faster than SGD throughout epochs.
Revised: Figureshows that SVI also converges faster than SGD throughout epochs.

Original: See Appendix B.2, Figure 9 for additional results that illustrates model intermediate convergence results.
Revised: See Appendix B.3, Figure 8 for additional results that illustrates model intermediate convergence results.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: E rr o r

Revised: SGDTraining

Revised: SGDTest

Revised: SVITraining

Revised: SVITest

Original: 5.1.2 Two-layer GCN model recovery.
Revised: 5.1.2 Two-layer GCN model recovery.

Original: The underlying graphs and data-generating process are described in Appendix B.2.
Revised: The underlying graphs and data-generating process are described in Appendix B.3.

Original: We aim to compare the model recovery performances measured by
Revised: We aim to compare the model recovery performances measured by

Original: Eq.
Revised: Eq.

Original: (15) for p = 2 , ∞ , when the true graph is known (i.e., edge sets fully known) or perturbed.
Revised: (16) for p = 2 , ∞ , when the true graph is known (i.e., edge sets fully known) or perturbed.

Original: We generate large random graphs and let N = N 1 = 2000 , B = 100 , E = 200 , lr =0.001, and momemtum = 0.99.
Revised: We generate large random graphs and let N = N 1 = 2000 , B = 100 , E = 200 , lr =0.001, and momemtum = 0.99.

Original: We also use the Nesterov momentum (Sutskever et al., 2013) when training the two-layer GCN model with ReLU activation.
Revised: We also use the Nesterov momentum (Sutskever et al., 2013) when training the two-layer GCN model with ReLU activation.

Original: Table 2 shows that SVI consistently reaches smaller ℓor ℓ ∞ model recovery error and MSE error.
Revised: Table 2 shows that SVI consistently reaches smaller ℓor ℓ ∞ model recovery error and MSE error.

Original: The pattern is consistent even if the graph information is perturbed.
Revised: The pattern is consistent even if the graph information is perturbed.

Original: Figure 3 shows that SVI consistently converges faster than SGD.[Table] [Figure] [Figure]
Revised: Figure 3 shows that SVI consistently converges faster than SGD.

Original: 5. R EAL - DATA E XPERIMENTS

Original: [Figure]
Revised: [Figure]

Original: 5.2.1 Binary solar ramping prediction. The data collection and description are described in Appendix B.3. The goal is to identify ramping events daily, abrupt changes in the solar power generation.
Revised: [Table]

Original: We train three-layer GCN models with lr = 0.001, momemtum = 0.99, and the Nesterov momentum. Table 3 shows that SVI almost always outperforms SGD under any metric on both training and test data. The pattern is consistent across different numbers of hidden neurons. Figure 5 also shows that
Revised: [Figure]

Original: SVI converges faster than SGD, which aligns with earlier simulation results. See Appendix B.3, Figure 13 (CA) and 14 (LA) for intermediate convergence results, including those under BN. [Table]
Revised: SVI reaches smaller error under faster initial convergence.

Original: [Figure]
Revised: [Figure]

Original: converges faster
Revised: 5. R EAL - DATA E XPERIMENTS

Original: 5.2.1 Multi-class traffic flow anomaly detection.
Revised: 5.2.1 Multi-class traffic flow anomaly detection.

Original: The data collection and description are described in
Revised: The data collection and description are described in

Original: Appendix B.3.
Revised: Appendix B.4.

Original: The goal is identify anomalous bi-hourly traffic flow observations.
Revised: The goal is identify anomalous bi-hourly traffic flow observations.

Original: We train three-layer
Revised: We train three-layer

Original: GCN models with lr = 0.001, momemtum = 0. [Figure] and the Nesterov momentum.
Revised: GCN models with lr = 0.001, momemtum = 0.99, and the Nesterov momentum.

Original: Table 4 shows that SVI reaches smaller training and test classification error for large hidden neurons and remains competitive when fewer neurons are used.
Revised: Table 3 shows that SVI reaches smaller training and test classification error for large hidden neurons and remains competitive when fewer neurons are used.

Original: In terms of weighted F 1 scores, SVI also reaches higher training and test scores in nearly all cases.
Revised: In terms of weighted F 1 scores, SVI also reaches higher training and test scores in nearly all cases.

Original: Figure 6 shows that SVI converges faster than SGD, which aligns with simulation results.
Revised: Figure 5 shows that SVI converges faster than SGD, which aligns with simulation results.

Original: See Appendix B.3, Figure 15 for intermediate convergence results.
Revised: See Appendix B.4, Figure 12 for intermediate convergence results

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: SVITrainingSVITest

Original: SVI converges faster than SGD.
Revised: SVI converges faster than SGD.

Original: 5.2.2 Vision datasets.
Revised: 5.2.2 Vision datasets.

Original: We compare SVI with gradient-based method on training FC networks (FCNet) and LeNet (LeCun et al., 1998).
Revised: We compare SVI with gradient-based method on training FC networks (FCNet) and LeNet (LeCun et al., 1998).

Original: The FCNet has four layers with 512 hidden nodes in each hidden layer.
Revised: The FCNet has four layers with 512 hidden nodes in each hidden layer.

Original: We examine performance on the MNIST dataset (LeCun, 1998) and the CIFAR-10 dataset (Krizhevsky et al., 2009), where the lr is fixed as 0.001 throughout training.
Revised: We examine performance on the MNIST dataset (LeCun, 1998) and the CIFAR-10 dataset (Krizhevsky et al., 2009), where the lr is fixed as 0.001 throughout training.

Original: We train the network for 10 epochs and fix batch size as 64, so each epoch contains 938 (resp.
Revised: We train the network for 10 epochs and fix batch size as 64, so each epoch contains 938 (resp.

Original: 782) effective iterations (i.e., mini-batches) on MNIST (resp.
Revised: 782) effective iterations (i.e., mini-batches) on MNIST (resp.

Original: CIFAR-10).
Revised: CIFAR-10).

Original: To stabilize the performance of SVI in later iterations, we sometimes use a warm-start technique—the estimates by SVI after the initial 20% iterations initialize parameters for gradient-based methods in the rest 80% iterations.
Revised: To stabilize the performance of SVI in later iterations, we sometimes use a warm-start technique—the estimates by SVI after the initial 20% iterations initialize parameters for gradient-based methods in the rest 80% iterations.

Original: We do so because SVI shows faster initial convergence but due to the skipping step, may lowers final accuracy.
Revised: We do so because SVI shows faster initial convergence but due to the skipping step, may lowers final accuracy.

Original: Table 5 shows results under both networks, where SVI -SGD outperforms SGD on both data and networks.
Revised: Table 4 shows results under both networks, where SVI -SGD outperforms SGD on both data and networks.

Original: SVI -Adam is competitive against Adam but the benefits are unclear; we expect this situation due to a lack of theoretical analyses (even for the last layer) under Adam, which belong to future work.
Revised: SVI -Adam is competitive against Adam but the benefits are unclear; we expect this situation due to a lack of theoretical analyses (even for the last layer) under Adam, which belong to future work.

Original: Nevertheless, the highest test accuracy by SVI upon fixing the dataset and network is competitive/higher than that by gradient-based methods.
Revised: Nevertheless, the highest test accuracy by SVI upon fixing the dataset and network is competitive/higher than that by gradient-based methods.

Original: Figure 7 further visualizes training and test accuracies along the entire training iterations and shows faster initial convergence by SVI . 
Revised: Figure 6 further visualizes training and test accuracies along the entire training iterations and shows faster initial convergence by SVI . [Figure] [Table]

Original: 5.2.3 Multi-class large-scale OGB node classification.
Revised: 5.2.3 Multi-class large-scale OGB node classification.

Original: We lastly demonstrate the applicability of
Revised: We lastly demonstrate the applicability of

Original: SVI on the large ogbn-arxiv graph provided by the Open Graph Benchmark (Hu et al., 2020; 2021).
Revised: SVI on the large ogbn-arxiv graph provided by the Open Graph Benchmark (Hu et al., 2020; 2021).

Original: The graph is much larger than earlier examples, and we also use wider and deeper models; more details are in Appendix B.4.
Revised: The graph is much larger than earlier examples, and we also use wider and deeper models; more details are in Appendix B.5.

Original: We optimize with SVI , SVI -Adam, SGD, and Adam, and train for a fixed E = 500 epochs.
Revised: We optimize with SVI , SVI -Adam, SGD, and Adam, and train for a fixed E = 500 epochs.

Original: Figure 8 shows that SVI or SVI -Adam yield comparable results to SGD [Table] [Figure] [Figure]
Revised: Figure 7 shows that SVI or SVI -Adam yield comparable results to SGD or

Original: [Figure] [Figure] [Figure] [Figure]
Revised: Adam.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure] or Adam. In addition, the bottom row shows clearly that SVI and SVI -Adam converge much faster than SGD and Adam during initial epochs.
Revised: In addition, the bottom row shows clearly that SVI and SVI -Adam converge much faster than SGD and Adam during initial epochs.

Original: We believe this faster initial convergence is particularly useful on large-scale experiments, where it is computationally demanding to train networks.
Revised: We believe this faster initial convergence is particularly useful on large-scale experiments, where it is computationally demanding to train networks.

Original: See 
Revised: See Appendix

Original: Appendix B.4 for results under different number of hidden neurons.
Revised: B.5 for results under different number of hidden neurons (cf.

Original: [Figure]
Revised: Figure 13) and under different choices of learning rate (cf. Table 6).

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have investigated how a new MVI approach can be useful in training neural networks, with strong prediction guarantees in special cases and competitive performance as SGD or Adam.
Revised: We have investigated how a new MVI approach can be useful in training neural networks, with strong prediction guarantees in special cases and competitive performance as SGD or Adam.

Original: In essence, the algorithm SVI “skips” computing derivative of the point-wise non-linearities.
Revised: In essence, the algorithm SVI “skips” computing derivative of the point-wise non-linearities.

Original: Although this skipping seems counter-intuitive at first glance, the theoretical justifications in the idealized theoretical settings of training the last-layer, assuming previous layers are fully known, provides partial justifications to the procedure.
Revised: Although this skipping seems counter-intuitive at first glance, the theoretical justifications in the idealized theoretical settings of training the last-layer, assuming previous layers are fully known, provides partial justifications to the procedure.

Original: On various synthetic and real-data examples, our SVI seems to improve efficiency in the early stage of training, which is useful when computational resources are constrained.
Revised: On various synthetic and real-data examples, our SVI seems to improve efficiency in the early stage of training, which is useful when computational resources are constrained.

Original: At present, the following tasks remain open.
Revised: At present, the following tasks remain open.

Original: Practically, testing the performance of SVI on training large models for more complex problems is worth studying.
Revised: Practically, testing the performance of SVI on training large models for more complex problems is worth studying.

Original: Theoretically, the guarantees only hold when all except the last layers of the network are known.
Revised: Theoretically, the guarantees only hold when all except the last layers of the network are known.

Original: Instead, it is important to relax and quantify the level of knowledge we need for these layers to better explain the previous-layer heuristics of SVI .
Revised: Instead, it is important to relax and quantify the level of knowledge we need for these layers to better explain the previous-layer heuristics of SVI .

Original: Application-wise, it is useful to address a wider range of problems in GNN, including edge and graph classification (Zhou et al., 2020).
Revised: Application-wise, it is useful to address a wider range of problems in GNN, including edge and graph classification (Zhou et al., 2020).

Original: We will explore them in the future.
Revised: We will explore them in the future.

Original: A P ROOFS
Revised: A P ROOFS

Original: Proof of Lemma 4.1.
Revised: Proof of Lemma 4.1.

Original: We first verify the monotonicity of F by considering vectorized parameters, where for a matrix A ∈ R m × n  vec ( A ) ∈ R mn by stacking vertically columns of A .
Revised: We first verify the monotonicity of F by considering vectorized parameters, where for a matrix A ∈ R m × n , vec ( A ) ∈ R mn by stacking vertically columns of A .

Original: Note that this vectorization is simply used to make sure the Euclidean inner product between F and Θ L is well-defined; the fundamental meaning of Θ L (e g., as the channel-mixing coefficient) remains unchanged.
Revised: Note that this vectorization is simply used to make sure the Euclidean inner product between F and Θ L is well-defined; the fundamental meaning of Θ L (e g., as the channel-mixing coefficient) remains unchanged.

Original: With an abuse of notation we use the same Θ L and Θ to denote the vectorized parameterand the corresponding parameter space. For any Θ 1 ,L , Θ 2 ,L ∈ Θ ,
Revised: With an abuse of notation, we use the same Θ L and Θ to denote the vectorized parameter

Original: [Equation]
Revised: [Equation]

Original: where κ is defined in (2).
Revised: where κ is defined in (2).

Original: The first equality uses the fact that the Y part is cancelled and the first inequality holds when ϕ L is continuously differentiable on its domain.
Revised: The first equality uses the fact that the Y part is cancelled and the first inequality holds when ϕ L is continuously differentiable on its domain.

Original: If ϕ L is only monotone on its domain, the the first inequality will only be greater than 0.
Revised: If ϕ L is only monotone on its domain, the the first inequality will only be greater than 0.

Original: We then verify the K 2 -Lipschitz continuity of F .
Revised: We then verify the K 2 -Lipschitz continuity of F .

Original: For any Θ 1 ,L , Θ 2 ,L ∈ Θ ,
Revised: For any Θ 1 ,L , Θ 2 ,L ∈ Θ ,

Original: [Equation]
Revised: [Equation]

Original: where K 2 has been defined.
Revised: where K 2 has been defined.

Original: We repeated used the Cauchy–Schwarz inequality and the last inequality relies on the assumption that ϕ L is K -Lipschitz continuous.
Revised: We repeated used the Cauchy–Schwarz inequality and the last inequality relies on the assumption that ϕ L is K -Lipschitz continuous.

Original: Note that given random samples { X 1 , . . .
Revised: Note that given random samples { X 1 , . . .

Original: , X N } , the quantities E X
Revised: , X N } , the quantities E X

Original: { λ min ( η ⊺ (  X∗ L ) η
Revised: { λ min ( η ⊺ ( ˆ X L ) η

Original: (  X∗ L )) } and E
Revised: ( ˆ X L )) } and E

Original: X {∥ η (  X∗ L ) ∥ 22 } can be empirically approximated by sample averages.
Revised: X {∥ η ( ˆ X L ) ∥ 22 } can be empirically approximated by sample averages.

Original: Lastly, when the true signal Θ ∗ L is the input to F ,

Original: [Equation]

Original: where we used the fact tht E

Original: [ Y | X ] =

Original: ϕ L ( η ( X ∗ L )Θ ∗ L ) .

Original: Remark A.1 (Remark for Lemma 4.1) .
Revised: Remark A.1 (Remark for Lemma 4.1) .

Original: For simplicity, we assume from now on the stronger condition that λ min ( η ⊺ (  X∗ L ) η
Revised: For simplicity, we assume from now on the stronger condition that λ min ( η ⊺ ( ˆ X L ) η

Original: (  X∗ L )) > 0 for any generic nonlinear feature  X∗ L .
Revised: ( ˆ X L )) > 0 for any generic nonlinear feature ˆ X L .

Original: In addition, because we are considering the last layer in classification, ϕ L is typically chosen as sigmoid or softmax, both of which are differentiable so λ min ( ∇ ϕ L ) exits.
Revised: In addition, because we are considering the last layer in classification, ϕ L is typically chosen as sigmoid or softmax, both of which are differentiable so λ min ( ∇ ϕ L ) exits.

Original: [-Note the property F (Θ ∗ L ) = 
-]
Revised: [-Note the property F (Θ ∗ L ) = 
-]

Original: implies that Θ ∗ L is a solution to VI [ F, Θ ] .

Original: In particular, if κ > 0 , it is the unique solution.

Original: As a result, Θ ∗ L can be efficiently solved to a high accuracy using F (Θ L ) under appropriated chosen γ in (5).

Original: Proof of Lemma 4.2.
Revised: Proof of Lemma 4.4.

Original: The proof employs classical techniques when analyzing the convergence of projection descent in stochastic optimization, which appear in (Juditsky & Nemirovsky, 2019,
Revised: The proof employs classical techniques when analyzing the convergence of projection descent in stochastic optimization, which appear in (Juditsky & Nemirovsky, 2019,

Original: Proposition 3.2).
Revised: Proposition 3.2).

Original: First, for any Θ L ∈ Θ ,
Revised: First, for any Θ L ∈ Θ ,

Original: [Equation]
Revised: [Equation]

Original: [ Jensen’s Inequality ]
Revised: [ Jensen’s Inequality ]

Original: [Equation]
Revised: [Equation]

Original: By the form of F , we then have that E X,Y {∥ F (Θ L ) ∥ 22 } ≤ 4 M 2 for any Θ L .
Revised: By the form of F , we then have that E X,Y {∥ F (Θ L ) ∥ 22 } ≤ 4 M 2 for any Θ L .

Original: Next, note that each ˆΘ ( t ) L is a deterministic function of Z N := { ( X i , Y i ) } N i =1 .
Revised: Next, note that each ˆΘ ( t ) L is a deterministic function of Z N := { ( X i , Y i ) } Ni =1 .

Original: Define the difference of estimation and its expected value as
Revised: Define the difference of estimation and its expected value as

Original: [Equation]
Revised: [Equation]

Original: As a result,
Revised: As a result

Original: [Equation]
Revised: [Equation]

Original: Proj

Original: [Equation]

Original: [ The projection is a contraction ]
Revised: [ The projection is a contraction ]

Original: [Equation]
Revised: [Equation]

Original: Taking expectation of both sides with respect to Z N yields
Revised: Taking expectation of both sides with respect to Z N yields

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows by noting that F 1: N is an unbiased estimator of F , which satisfies
Revised: where the last inequality follows by noting that F 1: N is an unbiased estimator of F , which satisfies

Original: [Equation]
Revised: [Equation]

Original: due to  F (Θ ∗ L )= 0 .
Revised: due to Assumption 4.3 on F (Θ ∗ L ) .

Original: Then, using triangle inequality yields the result.
Revised: Then, using triangle inequality yields the result.

Original: Lastly, we prove by induction that if we define R :=
Revised: Lastly, we prove by induction that if we define R :=

Original: (2 M 2 ) /κ 2 , γ t := 1 /κ ( t + 1) , we haved t ≤
Revised: (2 M 2 ) /κ 2 , γ t := 1 /κ ( t + 1) , we have

Original: R t + 1 .
Revised: [Equation]

Original: (Base case when t = 0 .)
Revised: (Base case when t = 0 .)

Original: Let B be the ∥ · ∥ 2 diameter of Θ (e.g., ∥ Θ 1 − Θ 2 ∥ 22 ≤ B 2 ∀ (Θ 1 , Θ 2 ) ∈ Θ .
Revised: Let B be the ∥ · ∥ 2 diameter of Θ (e.g., ∥ Θ 1 − Θ 2 ∥ 22 ≤ B 2 ∀ (Θ 1 , Θ 2 ) ∈ Θ .

Original: Denote Θ + L , Θ − L ∈ B to satisfy ∥
Revised: Denote Θ + L , Θ − L ∈ B to satisfy ∥

Original: Θ + L − Θ − L ∥
Revised: Θ + L − Θ − L ∥

Original: 22 = B 2 .
Revised: 22 = B 2 .

Original: By the definition of κ ,
Revised: By the definition of κ ,

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, the Cauchy-Schwarz inequality yields
Revised: Meanwhile, the Cauchy-Schwarz inequality yields

Original: [Equation]
Revised: [Equation]

Original: Thus, B ≤ 2 M/κ .
Revised: Thus, B ≤ 2 M/κ .

Original: As a result, B 2 / 2 ≤ 2 M 2 /κ 2 = R .
Revised: As a result, B 2 / 2 ≤ 2 M 2 /κ 2 = R .

Original: Because d 0 := ∥ ˆΘ (0) L − Θ ∗ L ∥ 22 ≤ B 2 ,
Revised: Because d 0 := ∥ ˆΘ (0) L − Θ ∗ L ∥ 22 ≤ B 2 ,

Original: [Equation]
Revised: [Equation]

Original: (The inductive step from t − 1 to t .)
Revised: (The inductive step from t − 1 to t .)

Original: Note that by the definition of γ t , κγ t = 1 / ( t + 1) ≤ 1 / 2 .
Revised: Note that by the definition of γ t , κγ t = 1 / ( t + 1) ≤ 1 / 2 .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: whereby the proof is complete by the definition of d t and R .
Revised: whereby the proof is complete by the definition of d t and R .

Original: Proof of Theorem 4.3.
Revised: Proof of Theorem 4.5.

Revised: Define

Revised: [Equation]

Revised: which uses the true parameter Θ ∗ L .

Original: Note that when p = 2 ,
Revised: Note that when p = 2 ,

Original: [Equation]
Revised: [Equation]

Revised: We now bound (a) and (b) separately

Revised: [Equation]

Revised: Bound of (a).

Original: We  can then use the bound on E ˆΘ ( T ) {∥ ˆΘ ( T ) − Θ ∥ 2 } from the previous lemma to complete the proof.
Revised: We have that [Equation] ) L ,X {∥ “ Θ ( T ) L − Θ ∗ L ∥ 2 } . We can then use the bound on E ˆΘ ( T ) {∥ ˆΘ ( T ) − Θ ∥ 2 } from the previous lemma to complete the proof.

Original: In addition, because p -norm is decreasing in p , we have that the bound holds for any p ∈ [2 , ∞ ] .
Revised: In addition, because p -norm is decreasing in p , we have that the bound holds for any p ∈ [2 , ∞ ] .

Original: Proof of Corollary 4.4. Under the definition thatL ∗ ( X t , Y t,i )
Revised: Bound of (b). We have by Assumptions 4.2 and 4.3 that

Original: [Equation]
Revised: [Equation]

Original: where the last inequality uses the fact that the function ln( a ) is monotonically increasing on [0 , 1] and that we use p = ∞ in Theorem 4.3 to guarantee that E “ Θ ( T ) L {∥ (cid:98) E [ Y t | X t ] − E [ Y t | X t ] ∥ ∞ } ≤ ϵ .

Original: Proof of Theorem 4.6.
Revised: Proof of Theorem 4.7.

Original: The crux is to bound the expected value of the norm of F evaluated at the stochastic OE estimate.
Revised: The crux is to bound the expected value of the norm of F evaluated at the stochastic OE estimate.

Original: This bound results from (Kotsalis et al., 2020, Theorem 3.8), where in general, for any Θ L ∈ Θ , the authors use the residual
Revised: This bound results from (Kotsalis et al., 2020, Theorem 3.8), where in general, for any Θ L ∈ Θ , the authors use the residual

Original: [Equation]
Revised: [Equation]

Original: as the termination criteria for the recurrence under a certain choice of the Bregman’s distance V ( a, b ) ; we let V ( a, b ) =
Revised: as the termination criteria for the recurrence under a certain choice of the Bregman’s distance V ( a, b ) ; we let V ( a, b ) =

Original: ∥ a − b ∥ 22 / 2 in our case.
Revised: ∥ a − b ∥ 22 / 2 in our case.

Original: The quantity N Θ (Θ L ) :
Revised: The quantity N Θ (Θ L ) :

Original: = { y ∈ R p |⟨ y, Θ ′ − Θ L ⟩ , ∀ Θ ′ ∈ Θ } denotes the normal cone of Θ at Θ
Revised: = { y ∈ R [Equation] |⟨ y, Θ ′ − Θ L ⟩ , ∀ Θ ′ ∈ Θ } denotes the normal cone of Θ at Θ

Original: L .
Revised: L .

Original: Then, under the assumptions on F and choices of step sizes, we can restate (Kotsalis et al., 2020, Theorem 3.8) in our special case as
Revised: Then, under the assumptions on F and choices of step sizes, we can restate (Kotsalis et al., 2020, Theorem 3.8) in our special case as

Original: [Equation]
Revised: [Equation]

Original: When we assume Θ is the entire space, N Θ (Θ L ) = { 0 } whereby res (ˆΘ ( R ) L ) =
Revised: When we assume Θ is the entire space, N Θ (Θ L ) = { 0 } whereby res (ˆΘ ( R ) L ) =

Original: ∥ F (ˆΘ ( R ) L ) ∥ 2 .
Revised: ∥ F (ˆΘ ( R ) L ) ∥ 2 .

Original: Thus, the result follows.
Revised: Thus, the result follows.

Original: Furthermore, recall the fact that for any matrix A ∈ R m × n and vectors x, x ′ ∈ R n , we have
Revised: Furthermore, recall the fact that for any matrix A ∈ R m × n and vectors x, x ′ ∈ R n , we have

Original: [Equation]
Revised: [Equation]

Original: where σ min ( A ) denotes the smallest singular value of A .
Revised: where σ min ( A ) denotes the smallest singular value of A .

Original: As a result, by letting A = η (  X∗ L ) , x = (cid:98) E
Revised: As a result, by letting A = η ( ˆ X L ) , x = (cid:98) E

Original: [ Y t | X t ] , x ′ =
Revised: [ Y t | X t ] , x ′ =

Original: E [ Y t | X t ] we have in expectation that
Revised: E [ Y t | X t ] we have in expectation that

Original: [Equation]
Revised: [Equation]

Original: where we used the fact F ( ˆ Θ ( R ) L )
Revised: where we used the fact F (ˆΘ ( R ) L )

Original: [Equation]
Revised: [Equation]

Original: E X { [Equation] ⊺ 
Revised: E X { η ⊺ ( ˆ X L )[ (cid:98) E

Original: [Equation] X [Equation]
Revised: [ Y t | X t ] − E

Original: [Equation] X t ]]] } .
Revised: [ Y t | X t ]]] } .

Original: Proof of Proposition 4.7.
Revised: Proof of Proposition 4.8.

Original: For notation simplicity, denote η := η L ( X L ) and θ :=
Revised: For notation simplicity, denote η := η L ( X L ) and θ :=

Original: Θ L .
Revised: Θ L .

Original: Meanwhile, for two vectors a, b ∈ R n , the notation a/b denotes the point-wise division.
Revised: Meanwhile, for two vectors a, b ∈ R n , the notation a/b denotes the point-wise division.

Original: 1 denotes a vector of all 1.
Revised: 1 denotes a vector of all 1.

Original: In addition, the binary and categorical cross-entropu losses are defined as:
Revised: In addition, the binary and categorical cross-entropy losses are defined as:

Original: [Equation]
Revised: [Equation]

Original: In (9), ϕ ( x ) = exp( x ) / (1 + exp { x } ) is the sigmoid function applied point-wise.
Revised: In (10), ϕ ( x ) = exp( x ) / (1 + exp { x } ) is the sigmoid function applied point-wise.

Original: In (10), e k is the k th standard basis vector in R F +1 and ϕ ( x ) =
Revised: In (11), e k is the k th standard basis vector in R F +1 and ϕ ( x ) =

Original: exp
Revised: exp

Original: ( x i ) / (cid:80) j exp( x j ) is the softmax function applied row-wise.
Revised: ( x i ) / (cid:80) j exp( x j ) is the softmax function applied row-wise.

Original: We first consider the binary cross-entropy loss L ( Y, ϕ L ( η ( X L )Θ L )) defined in (9).
Revised: We first consider the binary cross-entropy loss L ( Y, ϕ L ( η ( X L )Θ L )) defined in (10).

Original: Note that we have
Revised: Note that we have

Original: [Equation]
Revised: [Equation]

Original: Thus, the gradient with respect to θ (which is Θ L ) can be written as
Revised: Thus, the gradient with respect to θ (which is Θ L ) can be written as

Original: [Equation]
Revised: [Equation]

Original: Taking expectation thus yields the result.
Revised: Taking expectation thus yields the result.

Original: We next consider the categorical cross-entropy loss L ( Y, ϕ L ( η ( X L )Θ L )) defined in (10).
Revised: We next consider the categorical cross-entropy loss L ( Y, ϕ L ( η ( X L )Θ L )) defined in (11).

Original: Note that we have
Revised: Note that we have

Original: [Equation]
Revised: [Equation]

Original: Thus, the gradient with respect to θ can be written as
Revised: Thus, the gradient with respect to θ can be written as

Original: [Equation]
Revised: [Equation]

Original: where in the definition of F (Θ) in (4), Y = e Y ∈ R F +1 if Y belongs to more than 2 classes.
Revised: where in the definition of F (Θ) in (4) [Equation] Y = e Y ∈ R F +1 if Y belongs to more than 2 classes.

Original: Taking expectation thus yields the result.
Revised: Taking expectation thus yields the result.

Original: Proof of Proposition 4.8.
Revised: Proof of Proposition 4.9.

Original: Note that we are equivalently showing that there exists Θ ∗ L ′ such that
Revised: Note that we are equivalently showing that there exists Θ ∗ L ′ such that

Original: [Equation]
Revised: [Equation]

Original: Because Θ ∗ L ′ is the minimizer of ℓ 2 error
Revised: Because Θ ∗ L ′ is the minimizer of ℓ 2 error

Original: [Equation]
Revised: [Equation]

Original: it is clear that
Revised: it is clear that

Original: [Equation]
Revised: [Equation]

Original: Now,
Revised: Now,

Original: [Equation]

Original: Orthogonality of eigenvectors

Original: [Equation]
Revised: [Equation]

Original: Therefore, the loss under the true parameter Θ ∗ L is bounded above by ϵ ∥
Revised: Therefore, the loss under the true parameter Θ ∗ L is bounded above by δ

Original: Θ ∗ L ∥
Revised: ∥

Original: [Equation] so there exists a Θ ∗ L ′ that achieves no larger error.
Revised: Θ ∗ L ∥ 2 , so there exists a Θ ∗ L ′ that achieves no larger error.

Original: B A DDITIONAL E XPERIMENTS
Revised: B A DDITIONAL DETAILS

Original: Setup and comparison metrics are described in Appendix B.1.
Revised: Setup and comparison metrics are described in Appendix B.2.

Original: We describe dataset details and show additional results in Appendices B.2 onward, which illustrate the behavior of SVI and SGD during the entire training epoch. 
Revised: We describe dataset details and show additional results in Appendices B.3 onward, which illustrate the behavior of SVI and SGD during the entire training epoch. B. GNN NOTATION

Revised: Our notation of GNN models (i.e., graph filtering) is standard.

Revised: Suppose we have an undirected and connected graph G =

Original: B. S ETUP AND C OMPARISON M ETRICS
Revised: ( V , E , W ) , where V is a finite set of n vertices, E is a set of edges, and W ∈ R n × n is a weighted adjacency matrix that encodes node connections. Let I n denote an identity matrix of size n . Let D be the degree matrix of W and L g =

Original: Setup.
Revised: I n − D − 1 / 2 WD − 1 / 2 be the normalized graph Laplacian, which has the eigen-decomposition L g = U Λ U T . For a graph signal X ∈ R n × C with C input channels, it is then filtered via a function g Θ ( L g ) which acts on L g with channel-mixing parameters Θ ∈ R C × F for F output channels. Thus, the filtered signal X ′ =

Revised: g Θ ( L g ) X .

Revised: B. S ETUP AND C OMPARISON M ETRICS Setup.

Original: All implementation are done using PyTorch (Paszke et al., 2019) and
Revised: All implementation are done using PyTorch (Paszke et al., 2019) and

Original: PyTorch Geometric (Fey & Lenssen, 2019) (for GNN).
Revised: PyTorch Geometric (Fey & Lenssen, 2019) (for GNN).

Original: Models are trained with one Tesla
Revised: Models are trained with one Tesla

Original: P100-PCIE-16GB.
Revised: P100-PCIE-16GB.

Original: Total amount of computing time is not reported, as SVI only alters the update direction, which is identical to gradient-descent methods in special cases (see Proposition 4.7), so that
Revised: Total amount of computing time is not reported, as SVI only alters the update direction, which is identical to gradient-descent methods in special cases (see Proposition 4.8), so that

Original: SVI requires nearly identical computing resources as SGD.
Revised: SVI requires nearly identical computing resources as SGD.

Original: To ensure fair comparison, we carefully describe the experiment setup.
Revised: To ensure fair comparison, we carefully describe the experiment setup.

Original: In particular, the following inputs are identical to both SVI and SGD in each experiment.
Revised: In particular, the following inputs are identical to both SVI and SGD in each experiment.

Original: Data: (a) the size of training and test data (b) batch (batch size and samples in mini-batches).
Revised: Data: (a) the size of training and test data (b) batch (batch size and samples in mini-batches).

Original: Model: (a) architecture (e
Revised: Model: (a) architecture (e

Original: g., layer choice, activation function, hidden neurons) (b) loss function.
Revised: g., layer choice, activation function, hidden neurons) (b) loss function.

Original: Training regime: (a) parameter initialization (b) hyperparameters for optimizers (e
Revised: Training regime: (a) parameter initialization (b) hyperparameters for optimizers (e

Original: g., learning rate, momemtum factor, acceleration) (c) total number of epochs
Revised: g., learning rate, momemtum factor, acceleration) (c) total number of epochs

Original: In short, all except the way gradients are defined are kept the same for each comparison—our proposed SVI backpropagates gradients with respect to hidden input and transform the gradient back to parameter domain, whereas SGD do so with respect to parameters in each hidden layer.
Revised: In short, all except the way gradients are defined are kept the same for each comparison—our proposed SVI backpropagates gradients with respect to hidden input and transform the gradient back to parameter domain, whereas SGD do so with respect to parameters in each hidden layer.

Original: Comparison metrics.
Revised: Comparison metrics.

Original: For a random feature X ∈ R n × C , where n is the number of graph nodes and
Revised: For a random feature X ∈ R n × C , where n is the number of graph nodes and

Original: C is the input dimension, let the true (or predicted) model be E
Revised: C is the input dimension, let the true (or predicted) model be E

Original: [ Y | X, Θ] ∈ R n × F (or E [ Y | X, “ Θ] ), where F is the output dimension.
Revised: [ Y | X, Θ] ∈ R n × F (or E [ Y | X, “ Θ] ), where F is the output dimension.

Original: Given N realized pairs { ( X i , Y i ) } N i =1 where N denotes either the training or test sample size, we employ the following metrics for various tasks.
Revised: Given N realized pairs { ( X i , Y i ) } Ni =1 where N denotes either the training or test sample size, we employ the following metrics for various tasks.

Original: [Equation]
Revised: [Equation]

Revised: Cross-

Revised: entropy loss := N − 1 N (cid:88) i = n (cid:88)

Revised: j =

Revised: Y Ti,j E

Revised: [ Y i |

Revised: X i , “ Θ] j (13)

Revised: Classification error :

Revised: = ( n · N ) − 1 N (cid:88)

Revised: i = n (cid:88)

Revised: j =

Revised: F (cid:88) f

Revised: = 1 ( Y i,j,f ̸

Revised: = ˆ Y i,j,f ) (14) ℓ p parameter recovery error :

Revised: = ∥ ˆΘ − Θ ∥ p (15) ℓ p model recovery error :

Revised: = N − 1 N (cid:88) i = n (cid:88)

Revised: j = ∥

Revised: E [ Y i |

Revised: X i , “ Θ] j − E

Original:  For GCN model recovery We let p = 2 or ∞ in(14) and (15) and  when p = 2 , compute the relative error using ∥ Θ ∥ p or N − 1 (cid:80) N i =1 (cid:80)n
Revised: [ Y i | X i , Θ] j ∥ p . For GCN model recovery We let p = 2 or ∞ in (15) and (16) and when p = 2 , compute the relative error using ∥ Θ ∥ p or N − 1 (cid:80) Ni =1 (cid:80)

Original: j =1 ∥ E [ Y i |
Revised: nj =1 ∥ E [ Y i |

Original: X i , Θ] j ∥ p on the denominator.
Revised: X i , Θ] j ∥ p on the denominator.

Original: In addition, all results are averaged over three random trials, in which features X i are redrawn for simulated example and networks re-initialized.
Revised: In addition, all results are averaged over three random trials, in which features X i are redrawn for simulated example and networks re-initialized.

Original: We show standard errors in tables as brackets and in plots as error bars.
Revised: We show standard errors in tables as brackets and in plots as error bars.

Original: Partial results are shown due to space limitation, where Appendix B contains the extensive results.
Revised: Partial results are shown due to space limitation, where Appendix B contains the extensive results.

Original: B. T WO - LAYER FC NETWORKS AND GCN
Revised: B. T WO - LAYER FC NETWORKS AND GCN

Original: Graph data generation.
Revised: Graph data generation.

Original: Given a graph G =
Revised: Given a graph G =

Original: ( V , E ) , |V| = n and a signal X i ∈ R n × C , we generate
Revised: ( V , E ) , |V| = n and a signal X i ∈ R n × C , we generate

Original: Y i ∈ R n × F , where E [ Y i |
Revised: Y i ∈ R n × F , where E [ Y i |

Original: X i ] is a two/three-layer GCN model with ReLU (layer 1/layer 1 and 2) and sigmoid (layer 2/layer 3) activation.
Revised: X i ] is a two/three-layer GCN model with ReLU (layer 1/layer 1 and 2) and sigmoid (layer 2/layer 3) activation.

Original: We let C = 2 and F = 1 and let the true number of hidden nodes in each hidden layer always be 2.
Revised: We let C = 2 and F = 1 and let the true number of hidden nodes in each hidden layer always be 2.

Original: Entries of all true weight and bias parameters (resp.
Revised: Entries of all true weight and bias parameters (resp.

Original: features
Revised: features

Original: X i ) are i.i.d.
Revised: X i ) are i.i.d.

Original: samples from N (1 , 1) (resp.
Revised: samples from N (1 , 1) (resp.

Original: N (0 , 1) ) under a fixed seed (resp.
Revised: N (0 , 1) ) under a fixed seed (resp.

Original: fixed seeds per random trial).
Revised: fixed seeds per random trial).

Original: We consider both small ( n = 15 ) and large ( n = 40 ) graphs, where P [( i, j ) ∈ E ] = 0 .
Revised: We consider both small ( n = 15 ) and large ( n = 40 ) graphs, where P [( i, j ) ∈ E ] = 0 .

Original: Figure 10 visualizes the graphs.
Revised: Figure 9 visualizes the graphs.

Original: The true parameters are identical in both graphs.
Revised: The true parameters are identical in both graphs.

Original: To perturb the true graph, we perturb the edge set—a p fraction of edges in E and in E C are randomly discarded and inserted, where E C denotes edges in a fully-connected graph that does not exist in E .
Revised: To perturb the true graph, we perturb the edge set—a p fraction of edges in E and in E C are randomly discarded and inserted, where E C denotes edges in a fully-connected graph that does not exist in E .

Original: We set p = 0 .
Revised: We set p = 0 .

Original: 2 (resp. 0.05) for small (resp.
Revised: 2 (resp. 0.05) for small (resp.

Original: large) graphs.
Revised: large) graphs.

Original: Summary.
Revised: Summary.

Original: We briefly explain what each figure or table in this section contains.
Revised: We briefly explain what each figure or table in this section contains.

Original: Figure 9 shows model intermediate convergence results for the FC networks.
Revised: Figure 8 shows model intermediate convergence results for the FC networks.

Original: Figure 10 visualizes the small and large random graphs.
Revised: Figure 9 visualizes the small and large random graphs.

Original: Table 6 and Figure 11 shows additional results regarding model recovery on the small random graph.
Revised: Table 5 and Figure 10 shows additional results regarding model recovery on the small random graph.

Original: Figure 12 shows intermediate convergence results on both small and large random graphs. 
Revised: Figure 11 shows intermediate convergence results on both small and large random graphs. [Figure] [Figure] [Figure]

Original: performance
Revised: [Figure] [Figure] [Figure] estimation, comparison metrics, and whether graph is perturbed.

Original: Metrics are defined in (15) and (11) respectively
Revised: Metrics are defined in (16) and (12) respectively

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original:  consistently reaches smaller error under faster convergence, even just after the first training epoch.
Revised: SVI consistently reaches smaller error under faster convergence, even just after the first training epoch

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: B. T HREE - LAYER REALSOLAR OR TRAFFIC DATA
Revised: B. T HREE - LAYER REAL TRAFFIC DATA

Original: Solar data.

Original: The raw solar radiation data are retrieved from the National Solar Radiation Database for 2017 and 2018 https://nsrdb.nrel.gov/ .

Original: We consider data from 10 cities downtown in California and from 9 locations in downtown Los Angeles, where each city or location is a node in the network.

Original: The goal is to identify ramping events daily, abrupt changes in the solar power generation.

Original: [-Thus, Y t,i = 
-]
Revised: [-Thus, Y t,i = 
-]

Original: if node i at day t experiences a ramping event.

Original: We define feature

Original: X t :=

Original: { Y t − 1 . . .

Original: , Y t − d } as the collection of past d days of observation and pick d = 5 .

Original: We estimate edges via a k -nearest neighbor approach based on the correlation between training ramping labels, with k = 4 .

Original: Data in 2017 are used for training ( N = 360 ) and the rest for testing ( N 1 = 365 ), and we let B = 30 and E = 100 .

Original: We first show results for the Los-Angelas solar data, and then provide the complete results that demonstrate the behavior of both methods over epochs, without and with BN.

Original: [Table]

Original: No

Original: [Table]

Original: normalization

Original: With batch normalization

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: Traffic data.
Revised: Traffic data.

Original: The raw bi-hourly traffic flow data are from the California Department of Transportation, where we collected data from 20 non-uniformly spaced traffic sensors in 2020 https://pems.
Revised: The raw bi-hourly traffic flow data are from the California Department of Transportation, where we collected data from 20 non-uniformly spaced traffic sensors in 2020 https://pems.

Original: dot.ca.gov/ .
Revised: dot.ca.gov/ .

Original: Data are available hourly, with Y t,i = 1 (resp.
Revised: Data are available hourly, with Y t,i = 1 (resp.

Original: if the current traffic flow lies outside the upper (resp.
Revised: if the current traffic flow lies outside the upper (resp.

Original: lower) 90% quantile over the past four days of traffic flow of its nearest four neighbors based on sensor proximity.
Revised: lower) 90% quantile over the past four days of traffic flow of its nearest four neighbors based on sensor proximity.

Original: As before, we define feature X t as the collection of past d days of observation and set d = 4 , where the edges include the nearest five neighbors based on location.
Revised: As before, we define feature X t as the collection of past d days of observation and set d = 4 , where the edges include the nearest five neighbors based on location.

Original: Data in the first nine months are training data (e.g., N = 6138 ) and the rest for testing ( N 1 = 2617 ).
Revised: Data in the first nine months are training data (e.g., N = 6138 ) and the rest for testing ( N 1 = 2617 ).

Original: We let B = 600 and E = 100 and use the cross-entropy loss to test the performance of SVI under alternative loss functions.
Revised: We let B = 600 and E = 100 and use the cross-entropy loss to test the performance of SVI under alternative loss functions.

Original: [Table]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: SGDTest

Revised: SVITraining

Revised: SVITest

Original: B. F OUR - LAYER OGB REAL - DATA
Revised: B. F OUR - LAYER OGB REAL - DATA

Original: Data and model description.
Revised: Data and model description.

Original: The graph nodes are papers to be classified into categories and edges denote citation among papers; it has ∼ 170 thousand nodes, 1.16 million edges, 128-dimensional node features [Figure] and 40 node classes.
Revised: The graph nodes are papers to be classified into categories and edges denote citation among papers; it has ∼ 170 thousand nodes, 1.16 million edges, 128-dimensional node features, and 40 node classes.

Original: This graph is significantly larger than earlier examples.
Revised: This graph is significantly larger than earlier examples.

Original: We train four-layer GCN models with hidden nodes kept at 512, which are both wider and deeper than earlier models.
Revised: We train four-layer GCN models with hidden nodes kept at 512, which are both wider and deeper than earlier models.

Original: Additional results.
Revised: Additional results.

Original: We conduct further experiments when the number of hidden neurons in each hidden layer is reduced to 128 or 256.
Revised: We conduct further experiments when the number of hidden neurons in each hidden layer is reduced to 128 or 256.

Original: Figure 16 shows improved initial convergence by SVI or
Revised: Figure 13 shows improved initial convergence by SVI or

Original: SVI -Adam over the competitors, a pattern consistently observed earlier, and competitive overall accuracies.
Revised: SVI -Adam over the competitors, a pattern consistently observed earlier, and competitive overall accuracies.

Original: In particular, SVI is consistent under different number of hidden neurons whereas SGD performs clearly worse under 128 hidden neurons.
Revised: In particular, SVI is consistent under different number of hidden neurons whereas SGD performs clearly worse under 128 hidden neurons.

Original: [Figure] [Figure] [Figure] [Figure]
Revised: We also show in Table 6 that the performance of

Original: [Figure]
Revised: SVI is stable under different learning rates

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: layers. The setup is identical to Figure 8. It is clear that SVI orSVI -Adam still exhibitscomparable final accuracy and improved convergence at initial training stages under such variants.
Revised: SVI or -Adam still exhibits final accuracy and improved convergence at initial training stages under such variants.

Original: Inparticular,
Revised: In

Original: SGD is sensitive to the number of hidden neurons, as its final accuraciesunder 128 hiddenneurons areclearly poorerthan those by SVI and very different from SGD performance under 256 hidden neurons.
Revised: SGD is sensitive to the number of hidden neurons as its final accuracies 128 hidden are poorer those by SVI and very different from SGD performance under neurons

Original: In contrast, SVI or SVI -Adam performs more consistently
Revised: In contrast, SVI or SVI -Adam performs more consistently

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: rate, with the same observed improvement over gradient-based method.