From: ICLR.cc/2023/Conference/Paper3142/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper3142/Authors
title: Request for further feedback from Reviewer 9g7F
comment: We appreciate if the reviewer may let us know whether the follow-up responses addressed the concerns.

From: ICLR.cc/2023/Conference/Paper3142/Authors
title: Request for further feedback from Reviewer PUVE
comment: We appreciate if the reviewer may let us know whether the follow-up responses addressed the concerns.

From: ICLR.cc/2023/Conference/Paper3142/Authors
title: Request for further feedback from Reviewer 9kww
comment: We appreciate if the reviewer may let us know whether the follow-up responses addressed the concerns.

From: ICLR.cc/2023/Conference/Paper3142/Reviewer_Hgig
title: Thank you for the rebuttal
comment: I thank the authors for their rebuttal and clarifying my questions, and further would like to thank the authors for admitting epsilon relaxation of all the previous layers. I still think that analysis is a bit restrictive (i.e., only considering that most of the analysis is applied for the last layer and borrows heavily from previous result).

While the method does seem to perform better, I am afraid that the SGD baselines might not be appropriately tuned either— for eg. cifar10 accuracy for adam is too low (something that is pointed by other reviewers as well). Further, the need for fine-tuning for vision datasets further makes it hard for me to see any actual benefits. 

I am going to keep my score of weak-accept.

From: ICLR.cc/2023/Conference/Paper3142/Authors
title: Response to reviewer PUVE's concern
comment: We appreciate the further comments by the reviewer. We hope the theoretical analyses still shine insights into why SVI performs better than SGD, especially in the special case where SVI solves a convex problem but SGD does not. Meanwhile, we think the contribution of the work lies less on the theory side than on the method and experiment sides——as we explained in the introduction, no prior work explored how monotone variational inequality can be used to train general multi-layer networks. Our SVI algorithm is thus the first to do so as an alternative training scheme that in practice, also shows benefits on various models for different datasets.

Regarding the effect of hyper-parameter tuning, we refer the reviewer to our response to reviewer Hgig (i.e., reviewer 5) regarding the question “do the authors see convergence benefits even when SGD/Adam and the SVI counterparts are trained with larger learning-rates?” We showed that larger learning rates for SVI and SGD improve the performance of both, but SVI still shows empirical benefits. The empirical results are shown in Table 6 of the revised appendix.

Regarding training larger scale models, we would like to extend the approach to training ResNets on larger datasets. We do not have results yet and we acknowledge this as a limitation of the work.

From: ICLR.cc/2023/Conference/Paper3142/Reviewer_PUVE
title: Concern
comment: I was not concerned about this point: "Therefore, it is unclear why SGD would yield a sharper bound to that of SVI". I agree with this claim. But a paper on the theory side that meets the acception bar would be expected to have some result that is sharper than the baseline. I did not see it here. That is my concern.

As to empirical results, the concern is whether SGD or its variants may achieve better convergence empirically with parameter tuning, or on larger scale models.

From: ICLR.cc/2023/Conference/Paper3142/Authors
title: Further response to reviewer 9g7F
comment: We appreciate reviewer 9g7F’s further comments. However, we think there is a potential misunderstanding regarding the method vs. theory parts of our work. 

Method-wise, our Algorithm 1 applies SVI to training *all* layers of a multi-layer network, including a four-layer GNN model (cf. Figure 7). Thus, we consider feature learning in all our experiments using SVI. In other words, SVI is *not* just applied to training the last layer of a multi-layer network, where doing so would indeed ignore feature learning. As a result, we believe that the proposed SVI algorithm is relevant to neural network training, based on consistently improved empirical performances over SGD on a wide variety of data and models when considering feature learning.

On the other end, theoretically, our analyses only consider last layer training and thus do not consider feature learning. Further efforts in the future are needed to possibly match our analyses with those in the random feature network (e.g., [reference1]).

From: ICLR.cc/2023/Conference/Paper3142/Reviewer_9g7F
title: Post-Rebuttal
comment: I would like to thank the authors' response. I have also read other reviews.

However, it does not address my concerns. I do not believe this algorithm trained on the last layer has merits over GD on multilayer neural networks (although the authors added a new result that assumes $\epsilon$-closeness to the ground truth layer-by-layer, which is unfortunately unrealistic as well).  In addition, it does not compare with the random feature model and does not consider feature learning, so it is hard to say that this paper is relevant to neural network training. 

Therefore I do not think the paper clears the bar of a high-impact venue such as ICLR and would like to keep my original score. 

Reviewer 9g7F

From: ICLR.cc/2023/Conference/Paper3142/Reviewer_vGXo
title: Thank you for your response
comment: I thank the authors for their responses and improving the writing of the paper as suggested. I raise my score to 6 as the response addresses my comments on the writing.

From: ICLR.cc/2023/Conference/Paper3142/Authors
title: Further response to reviewer 2
comment: We thank the reviewer for the follow-up comments. We first emphasize that the theoretical guarantees of SVI are not directly comparable to those by SGD. First, in the special case (e.g., single-layer/last layer with $\epsilon$-approximate output from previous layers), using SVI yields a convex optimization problem, whereas the approach by SGD is typically non-convex (except in special cases). More details for the single-layer case can be found in (Juditsky & Nemirovsky, 2019). Therefore, it is unclear why SGD would yield a sharper bound to that of SVI. Second, in the general case (e.g., multi-layer network training), we do not yet have guarantees using SVI for model recovery, and we have highlighted this as a part of our future work. 

We acknowledge that more empirical studies are needed, as explained in future work. However, we want to emphasize that the advantage of SVI over SGD has been demonstrated on realistic examples using a reasonably complex model. Specifically, the OGB graph neural network training example (cf. Figure 7) uses a network with three hidden GCN layers of 512 hidden node each, so that the model has ~615 thousand model parameters. Regarding the dataset, it has nodal features in $\mathbb{R}^{128}$, nodal label belonging to one of 40 possible classes, ~169 thousand nodes, and ~1.17 million edges, so that we believe the dataset is more than a toy example.

From: ICLR.cc/2023/Conference/Paper3142/Authors
title: Further response to reviewer 1
comment: We thank the reviewer for providing such suggestions. Overall, the empirical significance of our approach is to show consistently improved computational efficiency in training multi-layer neural networks using SVI over using gradient-based methods, especially SGD. The significance in method lies in non-trivially extending the approach of (Juditsky & Nemirovsky, 2019) to train generic multi-layer networks. 

In terms of the background, we would like to emphasize that the background section has been extended significantly in the revised draft, where we highlight the novelty of (Juditsky & Nemirovsky, 2019). In short, the problem of parameter recovery for generalized linear model using SGD is non-convex except in special cases, and that work reduces the parameter recovery problem to a convex problem using monotone variational inequality.

In terms of the intuition and significance of our approach, especially with respect to  (Juditsky & Nemirovsky, 2019), we have followed the reviews by R3 and R4 to revise the description of our proposed SVI approach. More details are given in Section 3.2 of the revised draft.

From: ICLR.cc/2023/Conference/Paper3142/Reviewer_9kww
title: Acknowledgement
comment: Thanks for the reply. I have also read other reviews and I may say that I am still not sure of the significance of this paper yet. This may be partially because I am not familiar with the relevant areas, e.g., the work of (Juditsky & Nemirovsky, 2019). Nevertheless, I think the authors should focus more on the background and intuitions rather than the heavy theoretical details. 

Finally, I state that the comments of other reviewers may be more valuable, and please refer more to theirs.

From: ICLR.cc/2023/Conference/Paper3142/Reviewer_PUVE
title: Cannot see its being superior to SGD
comment: I am still not able to see why it is superior to the baseline SGD. Theoretically it does not give sharper bounds (either the generic case, in the special case with a neural networks or even in the case when the only last layer changes).

The better performance on a few toy models and data sets would not be enough to claim empirical advantage as well.

From: ICLR.cc/2023/Conference/Paper3142/Authors
title: Response to Reviewer 5
comment: Please see the response-to-all comment above for common questions asked by multiple reviewers.

**Q**: ``Most of the results in the paper, for make assumptions that are not justified in the text. It is unclear how reasonable many of the assumptions are. For example, for the analysis of the prediction of a GNN (Proposition 4.8) the authors assume that $L_g^+ = L_g^{'+}$. It would be helpful if the authors provide how unreasonable such assumptions are in line with the final results.''

**A**: Assumptions in the paper either are quoted from the single-layer MVI analyses [Juditsky \& Nemirovski, 2019] or consider Lipschitz continuity of the activation function and/or the pre-activation mapping. We note that many activation functions (e.g., ReLU, ELU, Tanh) are contractive and appropriate normalizations (e.g., batch normalization) would typically yield lipschitz continuous pre-activation. Regarding the assumption of Proposition 4.9 of the revised draft, we revised and relaxed it to only bound the spectral norm difference between $L_g$ and $L_g^{'}$. In Table 2, we show that when the graph edges are perturbed, the network under SVI estimates induce greater errors than that under SVI estimates using perfect knowlegde of the graph. Nevertheless, networks under SVI estimates always yield improved accuracy and faster convergence than GD counterparts. 

**Q**: ``In the empirical validation, while the authors vary the side of the networks and hidden neurons, how dos the performance of SVI and SGD/Adam compare when the hyper-parameters are varied. Esp for the rate of convergence, for real datasets do the authors see convergence benefits even when SGD/Adam and the SVI counterparts are trained with larger learning-rates?''

**A**: We appreciate this suggestion and test the effect of larger learning-rates on the GNN prediction example (cf. Figure 7). Table 6 in the revised appendix shows that under larger learning rates, both SVI and SGD converge faster than original results in the paper. In particular, SVI yields higher final test accuracies than gradient-based methods and SVI-SGD still shows improved initial convergence than SGD.



**Q**: ``The authors mention for the vision dataset, that SVI requires a warm start technique. Is SGD also warm-started for the final comparison?''

**A**: We remark that the warm start technique in SVI does not refer to warm starting with a pre-trained model. Instead, we realized that on vision datasets, training all convolutional layers with SVI until model convergence tends to deteriorate final accuracy of the trained network slightly; this is possibly due to the instability in final training when SVI ``skips'' derivative of the activation function. Therefore, we chose to train the LeNet with SVI for some initial epochs and then use the estimates to warm-start a gradient-descent based training scheme. In both cases, SVI and SGD train the same network initialized with the same set of random parameters. 

When training fully-connected networks or graph neural networks, we always use SVI to train the networks until convergence and do not adopt this warm start technique. Why SVI behaves differently during final stages will be investigated in the future.

From: ICLR.cc/2023/Conference/Paper3142/Authors
title: Response to Reviewer 4
comment: Please see the "Response to all reviewer" section on the top, where we have addressed the questions regarding writing of Section 3.2 and attempted to improve the theoretical analyses.

From: ICLR.cc/2023/Conference/Paper3142/Authors
title: Response to Reviewer 3
comment: Please see the response-to-all comment above for common questions asked by multiple reviewers.

**Q**: ``Even if we do not consider feature learning, the authors did not compare the theoretical results [Lemma 4.2,
Theorem 4.3] with gradient descent. (e.g., the standard training on the random feature model with gradient descent
(e.g., [reference1]))...Yehudai and Shamir. On the Power and Limitations of Random Features for Understanding Neural Networks. NeurIPS 2019.''

**A**: We appreciate the reviewer for suggesting this comparison. We remark that our Theorem 4.5 in the revised draft provides prediction accuracy in the last layer training, with error decaying at $\mathcal{O}(1/T)$ for $T$ iterations. This theoretical setting differs from the one considered by [Yehudai and Shamir, 2019], where they show a two-layer neural network with polynomial number of hidden neurons with high probability has better generalization capability over any polynomial predictor from certain function classes. The corresponding results for using SVI require more analayses in the future.

**Q**: ``Algorithm 1 has two branches: one using SVI, and another one using GD. What if the SVI branch (line 3--line5) was never executed? The authors did not show why the SVI branch is important compared with GD in theory or experiments.''.

**A**: If the SVI branch is never executed, Algorithm 1 is identical to stochastic gradient descent. The branch is included to allow greater flexibility in practice: one can choose to use SVI for training only certain layers, rather than training all layers as we did in experiments (i.e., only use the SVI branch). In experiments, we show that the SVI branch consistently improves the initial efficiency of network training with competitive final performance against GD. The complete theoretical analyses of using SVI to train multi-layer networks require future works.

From: ICLR.cc/2023/Conference/Paper3142/Authors
title: Response to reviewer 2
comment: Please see the response-to-all comment above for common questions asked by multiple reviewers.

**Q**: ``The advantages of faster convergence are only shown on limited empirical examples but lack theoretical support''

**A**: We acknowledge in the conclusion that ``practically, testing the performance of SVI on training large models for more complex problems is worth studying''. Nevertheless, we believe that the various numerical results on training fully-connected networks, convolutional networks, and graph neural networks on simulated and real-data examples have demonstrated the benefits of our proposed alternative neural network training scheme using SVI. The theoretical statements have also been revised to relaxed the condition on knowing all previous layers.

**Q**: ``Computational cost has not been saved by using the new method.''

**A**: We believe one can save total computation of training the same neural network: we consistently show that SVI yields improved initial convergence and competitive final performance as gradient-based methods. As a result, it takes much less training epochs when using SVI for the trained network to reach a desired accuracy. The computation time per epoch is nearly identical between SVI and gradient-descent based methods, because the computation of the update direction for parameters by SVI only ``skips'' the entry-wise derivative of the activation with respect to its input. The details of SVI procedure are described in Section 3.2.

From: ICLR.cc/2023/Conference/Paper3142/Authors
title: Response to reviewer 1
comment: Please see the response-to-all comment above for common questions asked by multiple reviewers.

**Q**: `` I did not know what the advantage of MVI is compared with vanilla SGD''

**A**: We apologize for this unclarity. We now emphasize this in the last sentence of the first paragraph of Section 5. Nevertheless, our numerical results also show improved performance of SVI against the Adam baseline (cf. Figure 7 on training GNN).

**Q**:``It is shown that MVI is equivalent to SGD when ignoring the derivative of elementwise activation function. This is quite strange since MVI can be seen as calculating an incorrect gradient. For the ReLU case, Remark 3.1 gives an intuition for why this can be reasonable. However, how about general activations (e.g., tanh)?''

**A**: Regarding equivalence of MVI to SGD, we remark that this equivalence only holds for classification problems using sigmoid activation for binary classification or softmax activation for multi-class classification under certain pre-activation function. This is true due to the particular structure of the activation function and the pre-activation, where we derive the equivalence in the proof of Proposition 4.8. In particular, we emphasize that the sigmoid case was first dervied in [Juditsky \& Nemirovski, 2019, end of Section 3.1.1], where our proposition extends this fact to the single-layer multi-class case.

Regarding intuition in Remark 3.1 for general activations, we do not have satisfactory answers yet, but want to remark that we have experimented with other activations (e.g., Tanh and Softplus), where the results (not shown) are similar to what we have presented in the paper. We aim to investigate the intuition for general activation more in the future, both theoretically and empirically.

**Q**: ``The theoretical results (Theorem 4.3 and Corollary 4.4) are hard to understand. The authors did not explain these results clearly. Also, the quantity has a complicated definition, and I do not know how different parameters influence its value.''

**A**: We apologize for the confusion, where Theorem 4.3 is now Theorem 4.5 in the revised draft. The takeaway for Theorem 4.5 is that we have the $O(1/T)$ decay, where $T$ is the number of projected GD iterations using the monotone operator as update direction for the last layer training. Regarding parameters defining $C_t$: $M$ is used to bound the norm of the operator, $K$ is the lipschitz constant of non-linearity, $\lambda_{\max}(\eta^T\eta)$ is the largest singular value of pre-activation $\eta$, and $\kappa$ intuitively explains how strongly convex $F$ is. The constant $C_t$ is larger when $M, K, \lambda_{\max}$ are larger and when $\kappa$ is smaller. We removed Corollary 4.4 in the new draft because the results are harder to establish under $\epsilon$-approximate estimate from previous $L-1$ layers.

**Q**: ``It is shown in Table 5 that the classification accuracy of SVI is better than vanilla SGD/Adam. However, on CIFAR-10 Adam only achieves 48.4\% accuracy, which is very strange to me. Can the authors detail the experimental settings?''

**A**: The experiment setting is described in Section 5.2.2. We remark that the 48.4\% accuracy of Adam on CIFAR-10 is due to the use of fully-connected layers, which takes in the flattened image with original dimension $(3,32,32)$. When switching the network layers to convolutional layers, the accuracies increase to 71.8\% (cf. Table 4, second to last entry of the second to last row). We acknowledge the possibility of improving FC network classification accuracy to 70\% [Lin et al., 2016] but did not adopt the improvement techniques therein in our experiments.

[Reference]: Zhouhan Lin et al., How far can we go without convolution: Improving fully-connected networks, ICLR 2016 workshop track.

From: ICLR.cc/2023/Conference/Paper3142/Authors
title: Response to all reviewers
comment: We thank all five reviewers for their constructive feedback and recognition of our work. Please see below for our response to all the comments and questions raised. We first response to questions raised by multiple reviewers before we provide pinpoint response to each reviewer's concerns. We have also addressed all minor places in the revision. All changes to the paper are marked in blue in the revised draft.

For convenience, we denote R1=Reviewer 9kww, R2=Reviewer PUVE, R3=Reviewer 9g7F, R4=Reviewer vGXo, and R5=Reviewer Hgig.

**Theoretical limitation.** R2---R5 expressed concerns regarding the theoretical statement in Section 4, where our analyses required the output from previous $L-1$ layers of an $L$-layer neural network to be fully known, which is rarely the case in practice. Inspired by R5, we instead relaxed the assumption to only assume $\epsilon$-approximate between the estimate from previous $L-1$ layers and the unobserved ground truth. The detailed assumption is expressed in Assumption 4.2. The statement in Theorem 4.5 of the revised draft then includes a constant factor of $\epsilon$, where the constant is defined using certain Lipschitz continuity parameters. Thus, the revised analyses better allow us to explain the performance of SVI in multi-layer network training. 

**Improvement of writing.** Several reviewers suggests improvements. These are (R1) make introduction more reader-friendly; (R3) better highlight the technical novelty compared with [Juditsky \& Nemirovski, 2019]; (R4) better explain the connection between single-layer MVI to SVI training for multi-layers. In response, we have revised accordingly: the introduction now includes more details regarding the single-layer MVI (which inspires our SVI) and a high-levle description of SVI; the algorithm section (cf. Section 3.2) carefully discusses the technical challenges of extending single-layer MVI to multi-layer case, before we illustrate how an equivalent view of single-layer MVI allows us to train multi-layer networks as in SVI.

From: ICLR.cc/2023/Conference/Paper3142/Reviewer_9kww
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposed an approach to train neural networks by using monotone variational inequality (MVI). The authors argued that they are easy to implement and computationally efficient. They gave some theoretical analysis for the prediction error bound of test loss. They also showed that MVI is related to the gradient descent. Their experiments demonstrated that MVI can be used in a variety of tasks and datasets, achieving competitive performance to commonly used SGD/Adam optimization algorithms.
strength_and_weaknesses: **Strength**:
The authors first investigate how MVI can be useful in neural network training, which seems to be novel. The theoretical results seem to be non-trivial, and the experiment section is comprehensive.

**Weaknesses**:
1. **Poor readability**. I have to say that the current paper is very hard to follow. I strive to read this paper carefully, but after spending several hours I still could not grasp the main result of this paper. In particular, I did not know what the advantage of MVI is compared with vanilla SGD. Moreover, I get confused in the following questions:
- It is shown that MVI is equivalent to SGD when ignoring the derivative of elementwise activation function. This is quite strange since MVI can be seen as calculating an incorrect gradient. For the ReLU case, Remark 3.1 gives an intuition for why this can be reasonable. However, how about general activations (e.g., tanh)?
- The theoretical results (Theorem 4.3 and Corollary 4.4) are hard to understand. The authors did not explain these results clearly. Also, the quantity $C_t$ has a complicated definition, and I do not know how different parameters influence its value. 

In particular, the introduction part is very dense (within only half a page) and did not contain enough background for readers to understand the topic in this paper. I may suggest the authors carefully polish the paper so that it can be more friendly to readers.

2. **Strange experimental results**. It is shown in Table 5 that the classification accuracy of SVI is better than vanilla SGD/Adam. However, on CIFAR-10 Adam only achieves 48.4% accuracy, which is very strange to me. Can the authors detail the experimental settings?

3. **Typos**:
- In algorithm 1, the notation $\{(X_i,Y_i)\}_{i=1}^N$ uses subscript $i$ to represent different sample while $g_l(X_l, \Omega_l)$ uses subscript $l$ to represent layers.
- In algorithm 1, what does $\sum_{x\in X_l} X_l$ mean?
clarity,_quality,_novelty_and_reproducibility: This paper is very hard to follow. See the above section for details.
summary_of_the_review: This paper is very hard to follow. I think the authors should polish the paper carefully so that it can be more friendly to readers. Also, I have turned down the level of confidence.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3142/Reviewer_PUVE
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors proposed an alternative method for training neural networks different from stochastic gradient methods. They have empirically shown that the proposed method is competitive to SGD or Adam, with partial theoretical justification. 
strength_and_weaknesses: - Strength: The authors show empirically SVI achieves faster initial convergence than gradient based method. 
- Weakness:
1. Compared to stochastic gradient methods, theories are only established for the case when previous layers are fixed. In such cases, the theorem established has not shown superior performance to stochastic gradient methods. 
2. The advantages of faster convergence are only shown on limited empirical examples but lack theoretical support. 
3. Computational cost has not been saved by using the new method. 
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly, proposes the novel algorithms. The reviewer is a bit skeptical whether the faster convergence on limited empirical experiments can be reproduced extensively on larger datasets and models. 
summary_of_the_review: The paper proposes a novel method for training neural network, but has not provided strong support, either theoretically or empirically, for the advantage of this method over traditional optimization algorithms. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper3142/Reviewer_9g7F
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper considered an alternative approach for training neural networks: instead of using stochastic gradient descent-based approaches, this paper proposes to use a monotone variational inequality approach. In particular, the authors reformulate the training for the last layer as a monotone variational inequality problem and provide convergence guarantees. The authors also tested their algorithms on synthetic and real data, and it seems that the performance of the proposed SVI algorithm is better than SGD.
strength_and_weaknesses: Strength:
1. The paper presents a new algorithm based on monotone variational inequality to train neural networks. 
2. The paper is generally well-written.

Weaknesses:
1. The theoretical analysis of the paper crucially relies on the training on the last layer. It avoids the difficulty of the feature learning process in deep neural networks. 

2. Even if we do not consider feature learning, the authors did not compare the theoretical results [Lemma 4.2, Theorem 4.3] with gradient descent. (e.g., the standard training on the random feature model with gradient descent (e.g., [reference1]))

[reference 1] Yehudai and Shamir. On the Power and Limitations of Random Features for Understanding Neural Networks. NeurIPS 2019.

3. Algorithm 1 has two branches: one using SVI, and another one using GD. What if the SVI branch (line 3--line5) was never executed? The authors did not show why the SVI branch is important compared with GD in theory or experiments.

4. What is the technical novelty compared with [Juditsky &Nemirovski, 2019]? The authors should discuss the fundamental difficulty of applying the techniques in  [Juditsky &Nemirovski, 2019] from the generalized linear model to general neural networks.
clarity,_quality,_novelty_and_reproducibility: Clarity: Good. The paper is generally well-written, and the contribution statement is clear.

Quality and Novelty: Medium. Although using SVI to solve neural networks appears new, the theory does not capture the feature learning in neural networks. In addition, the proposed algorithm still relies on gradient descent, and it is unclear to me what is the advantage of applying this method instead of SGD. Lastly, the authors did not explain why extending the techniques from [Juditsky &Nemirovski, 2019] to neural networks setting is challenging. 

Reproducibility: Good. The authors provided confidence intervals for experiments, and the code was also provided.
 

summary_of_the_review: The paper introduces an interesting approach of using monotone variational inequality to train neural networks. The theoretical analysis only allows training the last layer, and it is unclear whether this algorithm is better than predominating methods such as SGD or Adam in theory. It may only have a limited impact on the deep learning community.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: N/A
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3142/Reviewer_vGXo
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes stochastic variational inequality (SVI) as a new approach for training multi-layer neural networks. The SVI approach is inspired by the recent work of Juditsky & Nemirovsky which solves a monotone variational inequality (MVI) to fit a generalized linear model. The main idea in the SVI algorithm is to skip computing derivatives of the nonlinear activations in a neural network. Some theoretical results have been presented in Section 4 assuming that the weights of the layers before the last are given and fixed. The paper also discusses some numerical results which support the MVI training method. 
strength_and_weaknesses: Strengths:

1- The paper contains new ideas for training neural networks and provides numerical support for the proposed SVI approach.

Weaknesses:

1- The paper's writing and presentation need to be improved. The authors begin by discussing the MVI approach in (Juditsky & Nemirovsky, 2019), and in section 4 they try to connect the MVI approach to optimization in multi-layer deep learning. However, the connection remains unclear to me, and especially I cannot understand how the discussions in sections 3.1 and 3.2 are related. Section 3.2 presents Algorithm 1 before explaining how the algorithm follows from the MVI approach for linear models discussed in section 3.1, and the explanation in the paragraph of equation (6) was too vague to clarify the connection. I think the writing of Section 3.2 needs significant improvement.

2- The theoretical analysis in Section 4 assumes that the weights of all the layers before the last are given and fixed. It seems to me that this assumption simplifies the deep learning task to a fitting problem for a simple linear model. It is hard to see how the analysis can be extended to an actual multi-layer deep learning problem where the weights of all layers are assumed to be unknown. 
clarity,_quality,_novelty_and_reproducibility: Clarity: As mentioned in Weaknesses, I think the writing needs improvement, especially in section 3 where the authors need to better explain the MVI-based derivation of the training approach for multi-layer neural networks.

Quality: It seems to me that the connection between the MVI framework and Algorithm 1 for multi-layer neural networks is rather weak. Also, the paper's assumption of fixing the neural net's weights in all the layers before the last looks too strong. 

Originality: The proposed approach looks novel to me.
summary_of_the_review: This work attempts to apply an MVI approach to train a multi-layer neural network. However, the connection between the proposed algorithm and the MVI approach is unclear to me, and the theoretical guarantees are only applicable to a scenario where the weights of all layers before the last are given. Therefore, I think the paper's derivation of the framework and its theoretical analysis still need significant improvement.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3142/Reviewer_Hgig
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper introduces a new algorithm for training multi-layer neural networks which they refer to as Stochastic Variational Inequality, based on previous work by Judisky and Nemirosvsky (2019) on Monotone Variational Inequality. 

The algorithm treats the recovery of each layer of the neural network as a estimating the parameters of a GLM. This results in an algorithm results in a parameter update scheme in which differs from normal SGD based backpropagation such that it does not require the gradient w.r.t the non-linearity to be included in the chain rule. Therefore, the network can be trained normally using existing auto differentiation packages. 

The authors provide proof for the recovery of the last layer of the model parameters by following a sketch similar to Judisky and Nemirosvsky when the modulus of continuity is strictly greater than 0, and extend their analysis to the case when the modulus of continuity is equal to 0 (based on prior work by Kotsalis et al (2020)). 

The authors empirical compare SVI to SGD and its variats and demonstrate that for a given set of hyper-parameters SVI converges faster and has competitive if not better performance than SGD variants.
strength_and_weaknesses: The authors show through thorough experiments and theory that the monotone variational inequality based SVI algorithm to train NNs can gives a provide faster convergence (in some cases and tasks), along with theoretical guarantees on the recovery of the parameters. 

[Strengths]

- Empirically,  SVI based Adam and SGD get comparable or better performance than usual Adam and SGD, esp in terms of the rate of convergence in the initial epochs.

[Weaknesses]

- While the methodology is presented as that of the extension of MVI to multi-layer neural network, the analysis for the recovery of parameters is provided for a single (last layer), where it is assumed that final layer feature X is assumed to be “perfect”.  This makes it extremely similar to the setups in the previous work, where the authors study the recovery in GLMs.
- While I get that analyzing the recovery of all parameters of the network can be a bit cumbersome, can the results be extended to the case where the output of $L-1$ hidden layers is assumed to $\epsilon-$ approximate?
- Most of the results in the paper, for make assumptions that are not justified in the text. It is unclear how reasonable many of the assumptions are. For example, for the analysis of the prediction of a GNN (Proposition 4.8) the authors assume that L_g^+ = L’_g^+. It would be helpful if the authors provide how unreasonable such assumptions are in line with the final results.
- In the empirical validation, while the authors vary the side of the networks and hidden neurons, how dos the performance of SVI and SGD/Adam compare when the hyper-parameters are varied. Esp for the rate of convergence, for real datasets do the authors see convergence benefits even when SGD/Adam and the SVI counterparts are trained with larger learning-rates?
- The authors mention for the vision dataset, that SVI requires a warm start technique. Is SGD also warm-started for the final comparison?
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and is easy to follow.

The theoretical analysis uses techniques established in previous work, since the setup considered only analyzes the guarantees for the last layer of the neural network.
summary_of_the_review: The paper extends previous work on Monotone variational inequality and the analysis based on GLMs to general neural network based architectures, which results in an algorithm which “skips” the backprop step through the non-linearity $\phi$. The authors provide empirical compare their method to normal sgd based training to show the efficacy of their work. 

I am leaning towards weak-accept primarily due to the empirical validation of SVI against SGD and Adam. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold