We prove that for an L-layer fully-connected linear neural network, if the width of every hidden layer is Ω L · r · dout · κ 3 , where r and κ are the rank and the condition number of the input data, and dout is the output dimension, then gradient descent with Gaussian random initialization converges to a global minimum at a linear rate. The number of iterations to find an ǫ-suboptimal solution is O(κ log( 1 ǫ )). Our polynomial upper bound on the total running time for wide deep linear networks and the exp (Ω (L)) lower bound for narrow deep linear neural networks [Shamir, 2018] together demonstrate that wide layers are necessary for optimizing deep models. 1 We omit dependence on other parameters here. See Theorem 4.1 for the precise requirements.
REF shows that if the width of each layer is larger than Ω(L log(L)), then gradient descent converges to global minima at a rate O(log(1/ε)).
Width Provably Matters in Optimization for Deep Linear Neural Networks
ArXiv
ArXiv
Mathematics Computer Science