We prove new upper and lower bounds on the VC-dimension of deep neural networks with the ReLU activation function. These bounds are tight for almost the entire range of parameters. Letting W be the number of weights and L be the number of layers, we prove that the VC-dimension is O(W L log(W )) and Ω(W L log(W/L)). This improves both the previously known upper bounds and lower bounds. In terms of the number U of non-linear units, we prove a tight bound Θ(W U ) on the VC-dimension. All of these results generalize to arbitrary piecewise linear activation functions.
Harvey et al. prove upper and lower bounds on the VC-dimension of the hypothesis space of deep neural networks with the activation function of ReLU REF .
Nearly-tight VC-dimension bounds for piecewise linear neural networks
ArXiv
ArXiv
Mathematics Computer Science