We study the error landscape of deep linear and nonlinear neural networks with square error loss. We build on the recent results in the literature and present necessary and sufficient conditions for a critical point of the empirical risk function to be a global minimum in the deep linear network case. Our simple conditions can also be used to determine whether a given critical point is a global minimum or a saddle point. We further extend these results to deep nonlinear neural networks and prove similar sufficient conditions for global optimality in the function space. In this section, we describe the problem formulation and notations for deep linear neural networks, state main results (Theorems 2.1 and 2.2), and explain their implication. Suppose we have m input-output pairs, where the inputs are of dimension d x and outputs of dimension d y . Let X ∈ R d x ×m be the data matrix and Y ∈ R d y ×m be the output matrix. Suppose
Yun et al. REF extended these results and proposed sufficient and necessary conditions for a critical point to be a global minimum.
Global optimality conditions for deep neural networks
ArXiv
ArXiv
Mathematics Computer Science