We study the expressivity of deep neural networks with random weights. We provide several results, both theoretical and experimental, precisely characterizing their functional properties in terms of the depth and width of the network. In doing so, we illustrate inherent connections between the length of a latent trajectory, local neuron transitions, and network activation patterns. The latter, a notion defined in this paper, is further studied using properties of hyperplane arrangements, which also help precisely characterize the action of the neural network on the input space. We further show dualities between changes to the latent state and changes to the network weights, and between the number of achievable activation patterns and the number of achievable labelings over input data. We see that the depth of the network affects all of these quantities exponentially, while the width appears at most as a base. These results also suggest that the remaining depth of a neural network is an important determinant of expressivity, supported by experiments on MNIST and CIFAR-10. arXiv:1606.05336v2 [stat.ML] 24 Jun 2016 has been studied for neural networks [13, 14, 15] . In model distillation, more efficient models are trained to emulate costly models which possess better generalization error [16] . In this paper, we focus on the third of these properties, expressivity -the power of the class of deep neural networks to represent highly complex functions. We characterize how function complexity, under several natural and related definitions, grows with both width and depth, and use one of the definitions, the activation pattern of a deep network, to provide theory and intuition for how a deep network subdivides input space into regions. We verify our results experimentally on random networks, and in experiments on MNIST and CIFAR-10. The topic of neural network expressivity has a rich history, with different notions of expressivity studied. A particularly popular method of analysis has been examining achievable functions, the set of functions that are expressible by a fixed neural network architecture. This approach has yielded many fascinating results: neural networks have been shown to be universal approximators [17, 18] , their VC dimension studied [13, 14, 15] , and connections between boolean and threshold networks and ReLU networks developed in [19, 20] . VC dimension provides an upper bound on one of the properties we measure, and is further discussed in Appendix D.1. Looking at achievable functions also provides a method to compare different neural network architectures. A typical approach to this constructs classes of functions from one network architecture that another is incapable of approximating efficiently [21, 22, 23, 24] . This is often used to suggest the benefits of deep networks over shallow networks. Other work examines the complexity of a single function computed by a deep network, rather than the complexity of a set of functions. The number of linear regions was introduced as a measure of expressivity in [25] . In [26] , it was shown that a specific network achieves an exponential number of linear regions with depth. In Section 2.3 we generalize linear regions to network activation patterns, and develop a tight upper bound on the number of achievable activation patterns which the construction in [26] saturates. Many of these results, while compelling, also highlight a drawback of much of the existing work on expressivity -conclusions often rely on very particular functions or a specific weight setting of a neural network. In some sense, this misses the ultimate goal of understanding expressivity, which is to provide characteristic properties of networks and functions which apply for any typical task. Addressing this more general question requires determining a class of functions that are representative of the 'typical' or 'average' expressiveness of a particular architecture, and then understanding their salient features. We consider an average case which arises naturally in the practical usage of neural network architectures, the behavior of networks after random initialization. The expressivity of these random networks is largely unexplored, though a connection between random networks and compositional kernels is developed in [27] . The class of random networks enables tractable theoretical and experimental analysis of interrelated aspects of expressivity: trajectory length, neuron transitions, network activation patterns, and number of dichotomies (number of labelings on inputs). We find that all of these properties grow exponentially with depth, but only polynomially with width. Random data That we observe an exponential increase in these measures of expressivity, even when the network acts on random data, is particularly remarkable. A common explanation for the effectiveness of neural networks is that their functional form is well suited to the structure inherent in natural data, suggested empirically by e.g. the benefits of designing translation invariance into convolutional networks [28] , and explored theoretically in [29] . In particular the depth of the neural network is often seen to take advantage of the hierarchical and compositional structure of features in datasets [30] . It is surprising therefore that we see quantifiable advantages arising from deep neural networks even for unstructured data. Interrelated measures of expressivity All our measures of network expressivity are linked together through trajectory length. We demonstrate, both theoretically and experimentally, that the length of an input trajectory grows exponentially with the depth of a neural network. We then show
This highlights the need for new ways to analyze the capacity of a neural network (e.g., in terms of activation patterns or trajectory length REF ).
On the expressive power of deep neural networks
ArXiv
ArXiv
Computer Science Mathematics