Paper:
Nonlinear random matrix theory for deep learning
Neural network configurations with random weights play an important role in the analysis of deep learning. They define the initial loss landscape and are closely related to kernel and random feature methods. Despite the fact that these networks are built out of random matrices, the vast and powerful machinery of random matrix theory has so far found limited success in studying them. A main obstacle in this direction is that neural networks are nonlinear, which prevents the straightforward utilization of many of the existing mathematical results. In this work, we open the door for direct applications of random matrix theory to deep learning by demonstrating that the pointwise nonlinearities typically applied in neural networks can be incorporated into a standard method of proof in random matrix theory known as the moments method. The test case for our study is the Gram matrix
, where W is a random weight matrix, X is a random data matrix, and f is a pointwise nonlinear activation function. We derive an explicit representation for the trace of the resolvent of this matrix, which defines its limiting spectral distribution. We apply these results to the computation of the asymptotic performance of single-layer random feature networks on a memorization task and to the analysis of the eigenvalues of the data covariance matrix as it propagates through a neural network. As a byproduct of our analysis, we identify an intriguing new class of activation functions with favorable properties.

Review:
This looks like a good theoretical contribution and an interesting direction in the theory of deep learning to me.
In this paper, the authors compute the correlation properties (gram matrix) of the vector than went through some step of feedforward network with non linearities and random weights. Given the current interest in the theoretical description of neural nets, I think this is a paper that will be interesting for the NIPS audience and will be a welcome change between the hundreds of GAN posters. Some findings are particularly interesting.They applied their result to a memorization task and obtained an explicit characterizations of the training error. Also the fact that for some type of activation the eigenvalues of the data covariance matrix are constant  suggests interesting directions for the future.
Few comments and suggestions follows;:
- In section 1.3 : I think that the interest for Random Network is slightly older than the authors (maybe) know. In particular, these were standard setting in the 80s (more than 20 years before any work presented by the authors) and have generated a lot of attention, especially in the statistical physics community where the generalization properties were extensivly studied, (see, just to name a few:  -https://doi.org/10.1209/0295-5075/27/2/002 -https://doi.org/10.1088/0305-4470/21/1/030-10.1103/PhysRevLett.82.2975 )
- In connection with the random kitchen sinks: isn't the theory presented here a theory of what is (approximatly) going in in the Kernel space ? Given the connection between random projections and kernel methods there seems to be something interesting in this direction. Section  4.2 "Asymptotic performance of random feature methods" should propbably discuss this point. Perharps it is linked to the work of El Karoui, et al ?
- Can the author provided an example of an activation that leads to the 0 value for the parameter zeta? This is not clear from the definition (10) (or maybe I missed something obvious ?) is it just ANY even function?
I did not check all the computations in the supplementary materials (espcially the scary expansion in page 11!) but I am a bit confuse about the statement "t is easy to check many other examples which all support that eqn. (S19) is true, but a proof is still lacking." . Why is it called a lemma then? I am missing something?