Paper:
We propose a framework to understand the unprecedented performance and robustness of deep neural networks using field theory. Correlations between the weights within the same layer can be described by symmetries in that layer, and networks generalize better if such symmetries are broken to reduce the redundancies of the weights. Using a two parameter field theory, we find that the network can break such symmetries itself towards the end of training in a process commonly known in physics as spontaneous symmetry breaking. This corresponds to a network generalizing itself without any user input layers to break the symmetry, but by communication with adjacent layers. In the layer decoupling limit applicable to residual networks (He et al., 2015a), we show that the remnant symmetries that survive the non-linear layers are spontaneously broken based on empirical results. The Lagrangian for the non-linear and weight layers together has striking similarities with the one in quantum field theory of a scalar. Using results from quantum field theory we show that our framework is able to explain many experimentally observed phenomena, such as training on random labels with zero error (Zhang et al., 2017), the information bottleneck and the phase transition out of it and the following increase in gradient variances (Shwartz-Ziv & Tishby, 2017), shattered gradients (Balduzzi et al., 2017), and many more.

Review:
In this paper, an number of very strong (even extraordinary) claims are made:
* The abstract promises "a framework to understand the unprecedented performance and robustness of deep neural networks using field theory."
* Page 8 states that this is "This is a first attempt to describe a neural network with a scalar quantum field theory."
* Page 2 promises the use of the "Goldstone theorem" (no less) to understand phase transition in deep learning
* It also claim that many "seemingly different experimental results can be explained by the presence of these zero eigenvalue weights."
* Three important results are stated as "theorem", with a statement like "Deep feedforward networks learn by breaking symmetries" proven in 5 lines, with no formal mathematics.
These are extraordinary claims, but  when reaching page 5, one sees that the basis of these claims seems to be the Lagrangian of a simple phi-4 theory, and Fig. 1 shows the standard behaviour of the so-called mexican hat in physics, the basis of the second-order transition. Given physicists have been working on neural network for more than three or four decades, I am surprise that this would enough to solve all these problems!
I tried to understand these many results, but I am afraid I cannot really understand or see them. In many case, the explanation seems to be a vague analogy. These are not without interest, and maybe there is indeed something deep in this paper, but it is so far hidden by the hype. Still, I fail to see how the fact that phase transitions and negative direction in the landscape is a new phenomena, and how it explains all the stated phenomenology. Beside, there are quite a lot of things known about the landscape of these problems 
Maybe I am indeed missing something, but i clearly suspect the authors are simply overselling physics results.
I have been wrong many times, but I beleive that the authors should probably precise their claim, and clarify the relation between their results and both the physics AND statistics litterature, or better, with the theoretical physics litterature applied to learning, which is ---astonishing-- absent in the paper.
About the content:
The main problem for me is that the whole construction using field theory seems to be used to advocate for the appearence of a phase transition in neural nets and in learning. This rises three comments:
(1) So we really need to use quantum field theory for this? I do not see what should be quantum here (despite the very vague remarks page 12 "WHY QUANTUM FIELD THEORY?")
(2) This is not new. Phase transitions in learning in neural nets are being discussed since aboutn 40 years, see for instance all the pionnering work of Sompolinky et al. one can see for instance the nice review in https://arxiv.org/abs/1710.09553 In non aprticular order, phase transition and symmetry breaking are discussed in
* "Statistical mechanics of learning from examples", Phys. Rev. A 45, 6056 – Published 1 April 1992
* "The statistical mechanics of learning a rule", Rev. Mod. Phys. 65, 499 – Published 1 April 1993
* Phase transitions in the generalization behaviour of multilayer neural networks
http://iopscience.iop.org/article/10.1088/0305-4470/28/16/010/meta
* Note that some of these results are now rigourous, as shown in "Phase Transitions, Optimal Errors and Optimality of Message-Passing in Generalized Linear Models", https://arxiv.org/abs/1708.03395
* The landscape of these problems has been studied quite extensivly, see for instance "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization", https://arxiv.org/abs/1406.2572
(3) There is nothing particular about deep neural net and neural nets about this. Negative direction in the Hessian in learning problems appears in matrix and tensor factorizaion, where phase transition are well understood (even rigorously, see for instance, https://arxiv.org/abs/1711.05424 ) or in problems such as unsupervised learning, as e.g.:
https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.86.2174
https://journals.aps.org/pre/pdf/10.1103/PhysRevE.50.1766
Here are additional comments:
PAGE 1:
* "It has been discovered that the training process ceases when it goes through an information bottleneck (ShwartzZiv & Tishby, 2017)".
While this paper indeed make a nice suggestion, I would not call it a discovery yet as this has never been shown on a large network. Beside, another paper in the conference is claiming exacly the opposite, see : "On the Information Bottleneck Theory of Deep Learning". This is still subject of discussion.
* "In statistical terms, a quantum theory describes errors from the mean of random variables. "
Last time I studied quantum theory, it was a theory that aim to explain the physical behaviours at the molecular, atomic and sub-atomic levels, usinge either on the wave function (Schrodinger) or the Matrix operatir formalism (Hesienbger) (or if you want, the path integral formalism of Feynman).
It is certainly NOT a theory that describes errors from the mean of random variables. This is, i beleive, the field of "statistics" or "probability" for correlated variables. It is certianly used in physics, and heavily both in statistical physics and in quantum thoery, but this is not what the theory is about in the first place.
Beside, there is little quantum in this paper, I think most of what the authors say apply to a statistical field theory ( https://en.wikipedia.org/wiki/Statistical_field_theory )
* "In the limit of a continuous sample space, the quantum theory becomes a quantum field theory."
Again, what is quantum about all this? This true for a field theory, as well for continous theories of, say, mechanics, fracture, etc...
PAGE 2:
* "Using a scalar field theory we show that a phase transition must exist towards the end of training based on empirical results."
So it is a scalar classical field theory after all. This sounds a little bit less impressive that a quantum field theory. Note that the fact that phase transition arises in learning, and in a statistical theory applied to any learning process, is an old topic, with a classical litterature. The authors might be interested by the review "The statistical mechanics of learning a rule", Rev. Mod. Phys. 65, 499 – Published 1 April 1993
PAGE 8:
* "In this work we solved one of the most puzzling mysteries of deep learning by showing that deep neural networks undergo spontaneous symmetry breaking."
I am afraid I fail to see what is so mysterious about this nor what the authors showed about it. In any case, gradient descent break symmetry spontaneously in many systems, including phi-4, the Ising model or (in learning problems) the community detection problem (see eg https://journals.aps.org/prx/abstract/10.1103/PhysRevX.4.011047). I am afraid I miss what is new there...
* "This is a first attempt to describe a neural network with a scalar quantum field theory."
Given there seems to be little quantum in the paper, I fail to see the relevance of the statement. Secondly, I beleive that field theory has been used, many times and in greater lenght, both for statistical and dynamical problems in neural nets, see eg.
* http://iopscience.iop.org/article/10.1088/0305-4470/27/6/016/meta
* https://arxiv.org/pdf/q-bio/0701042.pdf
* http://www.lps.ens.fr/~derrida/PAPIERS/1987/gardner-zippelius-87.pdf
* http://iopscience.iop.org/article/10.1088/0305-4470/21/1/030/meta
* https://arxiv.org/pdf/cond-mat/9805073.pdf