Paper:
We investigate neural network training and generalization using the concept of stiffness. We measure how stiff a network is by looking at how a small gradient step on one example affects the loss on another example. In particular, we study how stiffness depends on 1) class membership, 2) distance between data points in the input space, 3) training iteration, and 4) learning rate. We experiment on MNIST, FASHION MNIST, and CIFAR-10 using fully-connected and convolutional neural networks. Our results demonstrate that stiffness is a useful concept for diagnosing and characterizing generalization. We observe that small learning rates reliably lead to higher stiffness at a given epoch as well as at a given training loss. In addition, we measure how stiffness between two data points depends on their mutual input-space distance, and establish the concept of a dynamical critical length that characterizes the distance over which datapoints react similarly to gradient updates. The dynamical critical length decreases with training and the higher the learning rate, the smaller the critical length.

Review:
This paper introduces the concept of stiffness: a measure of the change in the loss of sample A due to a gradient step based on sample B. It analyses the expected dynamic for A, B samples from the same and different classes, as well as, samples from the train and test sets.
To better understand the dynamics of optimization in neural networks is an open and important problem and the paper is clearly motivated in this regard. The proposed method is straight forward and I am not aware of a similar method. 
In addition to that, the paper also introduces "dynamical critical length ξ" which is the stiffness of A, B samples based on the cosine similarity of the respective inputs (section 2.4). A linear estimator of when this length becomes 0 is also introduced. Confusingly this is also called the "dynamical critical length ξ" in section 4.2. Later on the term "dynamical scale ξ" and "dynamical critical scale ξ" seem to be used interchangeably. Figure 6 mentions the "critical length χ" on the y-axis which seems to be a typo as no such measure was introduced.
The equivalence between eq. 2 and the two parts of eq. 3 is not obvious. We'd appreciate if the authors would provide a proof of such. 
Overall, the paper is written in a simple language but paragraphs remain surprisingly hard to understand. An example of such is e.g. section 4.4: What do the authors mean by "characteristic distance" between two input points? What is "the typical scale of spatial variation" of a function? etc.
The paper concludes that:
1.) there is a link between generalization and stiffness
2.) stiffness decreases with the onset of overfitting
3.) "general gradient updates with respect to a member of a class help to improve loss on data points in the same class"
4.) "The pattern breaks when the model starts overfitting to the training set, after
which within-class stiffness eventually reaches 0"
5.) This is observed for different models on different datasets
6.) "we observed that the farther the datapoints and the higher the epoch of training, the less
stiffness exists between them on average"
7.) "the higher the learning rate, the smaller the ξ"
Verdict: Reject
The conclusions are self-evident. The paper fails to demonstrate the usefulness of stiffness and most results are expected and provide little to no insights into the optimization dynamics of deep neural networks. In fact, the reasoning in this paper is almost tautological (conclusions 1-6).
E.g. if the A, B samples used to compute stiffness are separately drawn from the train and test set then stiffness is a proxy for the difference between the train error and the test error after another gradient step. The authors then compute stiffness at different points of the optimization procedure and conclude that stiffness decreases when the network starts to overfit. Since overfitting is the point in training where train error and test error diverge it is obvious that this can also be observed with regards to "stiffness". Hence, the reasoning is circular.
Conclusion 7 is slightly different in that it observes that larger learning rates result in smaller ξ which, given the previous paragraph, we can rewrite into the statement "larger learning rates generalise better". This is a well known empirical observation and has been discussed thoroughly (e.g. on connection with flat and sharp minima or learning rate decay schedules). 
Disclaimer: This review was done on short notice.