Paper:
For a long time, designing neural architectures that exhibit high performance was considered a dark art that required expert hand-tuning. One of the few well-known guidelines for architecture design is the avoidance of exploding or vanishing gradients. However, even this guideline has remained relatively vague and circumstantial, because there exists no well-defined, gradient-based metric that can be computed before training begins and can robustly predict the performance of the network after training is complete. We introduce what is, to the best of our knowledge, the first such metric: the nonlinearity coefficient (NLC). Via an extensive empirical study, we show that the NLC, computed in the network's randomly initialized state, is a powerful predictor of test error and that attaining a right-sized NLC is essential for attaining an optimal test error, at least in fully-connected feedforward networks. The NLC is also conceptually simple, cheap to compute, and is robust to a range of confounders and architectural design choices that comparable metrics are not necessarily robust to. Hence, we argue the NLC is an important tool for architecture search and design, as it can robustly predict poor training outcomes before training even begins.

Review:
This paper proposes a metric to measure the "nonlinearity" of neural network, and presents evidence that the value of this metric at initialization time is predictive of generalization performance.
Apart from a few problems I think this paper is well written and thorough. The contribution is solid, although not earth shattering given previous work on such metrics.  There seems to be a basic error in some of the early math, although I don't think this will qualitatively affect the results in any significant way.
-----------------
Detailed comments by section:
------------------
Section 3:
It seems like a 1/sqrt(d) factor is missing from these Q_i(S_x x(i)) and Q_j(S_x f(x,j)) formulas.  As far as I can tell this doesn't affect Def 1 because you seemed to use the correct formula there. 
However, the rewritten version with the traces doesn't seem to be correct. There should be a d_in factor in the denominator (inside the square root). This error seems unrelated to the other one.  Assuming I'm correct and that this is an error, does this affect your results in the various figures?  And what is the actual final definition of NLC that you used?
In general, it's annoying for the reader to verify that all of these forms are equivalent.  And it's fiddly enough with the sqrt(d) terms constantly disappearing and reappearing in the numerator and denominators that even you made multiple errors (as far as I can tell).  I would suggest making this section more rigorous and writing out everything carefully. And you probably don't need to rewrite it in so many equivalent forms with different notation unless they are useful somehow. 
The use of the Q and S symbols feels superfluous and counterproductive. Standard notation with expectations and squares wouldn't take much more space and would be a lot clearer.
Section 4:
"we plot the relative diameter of the linearly approximable regions of the network as defined in section 3": but you don't seem to define "relative diameter" there. As far as I can tell it's only defined in Appendix E, and this is only mentioned in the caption of figure 1.  It's impossible to interpret this result without knowing precisely what "relative diameter" is.  If you can't afford to describe this in the main paper you should at least mention that it's a different (more expensive) way of estimating the same thing that the NLC estimates.
In Figure 2, are the higher test errors due to the optimizer failing to lower the training error, or due to a greater generalization gap?  I guess the Figure 3 results suggest the latter possibility, which is surprising to me. 
What does it mean to have a "very biased output".  What does that inequality mean intuitively?  Should there be an absolute value on the RHS?  It would be much easier to parse it if it were written in plain notation without these S and Q symbols.
Section 6:
"metric also an" -> "metric also has an"
Can you generate a failure case for "correlation information" that doesn't involve Batch Norm layers?  I don't think the authors of those works meant for their results to deal with that.
Note that there are actually a lot of papers going back to the 90s that discussed and proved representational benefits of depth in neural networks.