We demonstrate that in residual neural networks (ResNets) dynamical isometry is achievable irrespectively of the activation function used. We do that by deriving, with the help of Free Probability and Random Matrix Theories, a universal formula for the spectral density of the input-output Jacobian at initialization, in the large network width and depth limit. The resulting singular value spectrum depends on a single parameter, which we calculate for a variety of popular activation functions, by analyzing the signal propagation in the artificial neural network. We corroborate our results with numerical simulations of both random matrices and ResNets applied to the CIFAR-10 classification problem. Moreover, we study the consequence of this universal behavior for the initial and late phases of the learning processes. We conclude by drawing attention to the simple fact, that initialization acts as a confounding factor between the choice of activation function and the rate of learning. We propose that in ResNets this can be resolved based on our results, by ensuring the same level of dynamical isometry at initialization. Deep Learning has achieved unparalleled success in fields such as object detection and recognition, language translation and speech recognition [2] . At the same time, models achieving these state-of-the-art results are increasingly deep and complex [3] , which often leads to optimization challenges such as vanishing gradients. Many solutions to this problem have been proposed. In particular, Residual Neural Networks remedy this problem to some extent [5, 7] by using skip connections in the network architecture, which improve gradient flow. As a result, Residual Neural Networks outmatched other competing models in the 2015 ILSVRC and COCO competitions. Yet another approach towards solving this problem is to tailor fit the networks weight initialization to facilitate training, for example by ensuring dynamical isometry [6] . In this latter case, the insights are based on an analysis of the statistical properties of information propagation in the network and a study of the full singular spectrum of a particular matrix, namely the input-output Jacobian, via the techniques of Free Probability and Random Matrix Theories (FPT & RMT). This perspective has recently led to successfully training a 10000 layer vanilla convolutional neural network [8] . RMT is a versatile tool that, since its inception, saw a substantial share of applications, form the earliest in nuclear physics [9] to the latest in game theory [10] (see [11] for some of the use cases discovered in the mean time). It is thus not surprising that it found its way to be used to understand artificial neural networks. In particular, to study their loss surface [12, 13] , the associated Gram matrix [14, 15] and in the case of single layer networks, their dynamics [16] . Our main contribution is extending the theoretical analysis of [6, 17, 18] to residual networks. In particular, we find that in contrast to feedforward networks, residual networks can obtain dynamical isometry for many different activation functions. These theoretical results are supported by an empirical investigation on the popular CIFAR-10 benchmark. For a short discussion of relevant results in pure RMT and FPT we relegate the reader to subsection II A. Here, we focus on particular developments related to our work on neural networks. The framework of dynamical mean field theory, we will apply to study signal propagation in neural networks, was first used in this context in [20] . There, the authors showed the existence of an order-to-chaos expressivity transition for deep feed forward neural networks with random initial weights, on the plane spanned by the variances of the network weights and biases. This in turn led to the insight of [17] , that arbitrary deep networks can be trained as long as they are close to the criticality associated with that transition. The techniques developed in these works, together with methods of FPT and RMT, allowed, for the first time, to analytically compute the singular value distribution of the input-output Jacobian of a deep feed forward network with nonlinear activation function and at criticality [6] . Finally, the authors of [18] showed, that for feed-forward neural networks, in their large depth limit and at a special point of the above mentioned * Electronic address: wojciech.tarnowski@uj.edu.pl † Electronic address:
In REF , the authors discuss the universal characters of the singular spectrum under the O(1/L) scaling with free probability.
Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function
AISTATS 2019
Computer Science Mathematics