Paper:
As deep neural networks (DNNs) achieve tremendous success across many applica-1 tion domains, researchers tried to explore in many aspects on why they generalize 2 well. In this paper, we provide a novel perspective on these issues using the gradient 3 signal to noise ratio (GSNR) of parameters during training process of DNNs. The 4 GSNR of a parameter is defined as the ratio between its gradient's squared mean and 5 variance, over the data distribution. Based on several approximations, we establish 6 a quantitative relationship between model parameters' GSNR and the generaliza-7 tion gap. This relationship indicates that larger GSNR during training process leads 8 to better generalization performance. Moreover, we show that, different from that 9 of shallow models (e.g. logistic regression, support vector machines), the gradient 10 descent optimization dynamics of DNNs naturally produces large GSNR during 11 training, which is probably the key to DNNs' remarkable generalization ability.

Review:
In this work, the authors suggest a new point of view on generalization through the lens of the distribution of the per-sample gradients. The authors consider the variance and mean of the per-sample gradients for each parameter of the model and define for each parameter the Gradient Signal to Noise ratio (GSNR). The GSNR of a parameter is the ratio between the mean squared of the gradient per parameter per sample (computed over the samples) and the variance of the gradient per parameter per sample (also computed over the samples). The GSNR is promising as a measure of generalization and the authors provide a nice leading order derivation of the GSNR as a proxy for the measure of the generalization gap in the model. After the derivation, experimental results on MNIST are presented and suggest that empirically there is a relation between the generalization gap of neural network trained by gradient descent and the GSNR quantity given in the paper. Next the author analyze the GNSR of DNNs as opposed with shallow models or other learning techniques and observe that the GSNR differs when using random labels (lower GSNR) as compared with true labels and exhibits different behavior along training for DNNs and gradient descent.
 Assumption 2.3.1 is not necessarily the most realistic in the current training paradigm since multi-epoch training indeed separates the training and testing per-sample gradient distributions significantly. 
Overall the paper gives a fresh (as far as I know) and nice idea on generalization of neural networks. The derivation requires quite a few stringent assumption (the leading order analysis, on the step size, assumption 2.3.1 on the test and train gradient distributions) the experiments do suggest that the theory is valid to an extent, especially during the early parts of training. In contrast, the presentation of the paper distracts from the work and needs additional cleaning up. Also, the experimental section 2.4 would benefit additional empirical analysis on other datasets  and additional experiments, as well as more thorough explanation of the experiments. The writing of the paper needs additional proofreading as currently it is easy to spot typos and grammar errors along the paper. I currently vote weak reject, for solid content and not-quite-ready presentation.
Additional experiments and careful proofreading should definitely enhance the paper and get it to the level of publications, so I am willing to change my decision if the authors improve the writing and the overall presentation. I like the idea presented in the paper and encourage the authors to resubmit a more tidy draft.
smaller presentation issues and typos/grammar issues:
Figure 1: name X and Y labels with more meaningful names instead of RHS,LHS
Figure 3(c): add legend
line 64: consists >>> which consists
line 73: we refer R(z) >>> we refer to R(z)
line 79 futher >>> further overall sentence needs more work
line 92 distribution becomes >>> distributions become 
line 100: using common notation summing over n samples, the sum should possibly start from i=1 to n instead of i = 0 to n
line 132 include >>> includes (also possibly rephrase sentence)
line 136 vefity >>> verify 
line 137: M number >>> M  
line 157 the data points closely distributed >>> the data points are closely distributed
line 162 thorough analytically >>> thorough analytical 

-------- Update After Rebuttal --------
I have read the comments the authors made and reviewed the paper's revision. 
The presentation has improved a lot (Even though there is still room to go). 
Nonetheless at this time I choose to update my score to a weak accept, as I think the authors bring a fresh (as far as I know) and interesting idea with regards to empirical generalization.