We prove that for an L-layer fully-connected linear neural network, if the width of every hidden layer is Ω L · r · dout · κ 3 , where r and κ are the rank and the condition number of the input data, and dout is the output dimension, then gradient descent with Gaussian random initialization converges to a global minimum at a linear rate. The number of iterations to find an ǫ-suboptimal solution is O(κ log( 1 ǫ )). Our polynomial upper bound on the total running time for wide deep linear networks and the exp (Ω (L)) lower bound for narrow deep linear neural networks [Shamir, 2018] together demonstrate that wide layers are necessary for optimizing deep models. 1 We omit dependence on other parameters here. See Theorem 4.1 for the precise requirements.
Most relevant to our work is the result of REF , which shows that if the width of hidden layers is larger than the depth, gradient descent with Gaussian initialization can efficiently converge to a global minimum.
Width Provably Matters in Optimization for Deep Linear Neural Networks
ArXiv
ArXiv
Mathematics Computer Science