We analyze the convergence of (stochastic) gradient descent algorithm for learning a convolutional filter with Rectified Linear Unit (ReLU) activation function. Our analysis does not rely on any specific form of the input distribution and our proofs only use the definition of ReLU, in contrast with previous works that are restricted to standard Gaussian input. We show that (stochastic) gradient descent with random initialization can learn the convolutional filter in polynomial time and the convergence rate depends on the smoothness of the input distribution and the closeness of patches. To the best of our knowledge, this is the first recovery guarantee of gradient-based algorithms for convolutional filter on non-Gaussian input distributions. Our theory also justifies the two-stage learning rate strategy in deep neural networks. While our focus is theoretical, we also present experiments that illustrate our theoretical findings. Deep convolutional neural networks (CNN) have achieved the state-of-the-art performance in many applications such as computer vision [Krizhevsky et al., 2012] , natural language processing [Dauphin et al., 2016] and reinforcement learning applied in classic games like Go [Silver et al., 2016]. Despite the highly nonconvex nature of the objective function, simple first-order algorithms like stochastic gradient descent and its variants often train such networks successfully. On the other hand, the success of convolutional neural network remains elusive from an optimization perspective. When the input distribution is not constrained, existing results are mostly negative, such as hardness of learning a 3-node neural network [Blum and Rivest, 1989] or a non-overlap convolutional filter [Brutzkus and Globerson, 2017] . Recently, Shamir [2016] showed learning a simple one-layer fully connected neural network is hard for some specific input distributions. These negative results suggest that, in order to explain the empirical success of SGD for learning neural networks, stronger assumptions on the input distribution are needed.. Recently, a line of research [Tian, 2017 , Brutzkus and Globerson, 2017 , Li and Yuan, 2017 , Soltanolkotabi, 2017 , Zhong et al., 2017 assumed the input distribution be standard Gaussian N (0, I) and showed (stochastic) gradient descent is able to recover neural networks with ReLU activation in polynomial time. * This work is done while the author is at Facebook AI Research.
REF extend the result to non-Gaussian input distributions with ReLU activation.
When is a Convolutional Filter Easy To Learn?
ArXiv
ArXiv
Computer Science Mathematics