We establish theoretical results concerning all local optima of various regularized Mestimators, where both loss and penalty functions are allowed to be nonconvex. Our results show that as long as the loss function satisfies restricted strong convexity and the penalty function satisfies suitable regularity conditions, any local optimum of the composite objective function lies within statistical precision of the true parameter vector. Our theory covers a broad class of nonconvex objective functions, including corrected versions of the Lasso for error-in-variables linear models; regression in generalized linear models using nonconvex regularizers such as SCAD and MCP; and graph and inverse covariance matrix estimation. On the optimization side, we show that a simple adaptation of composite gradient descent may be used to compute a global optimum up to the statistical precision ǫ stat in log(1/ǫ stat ) iterations, which is the fastest possible rate of any first-order method. We provide a variety of simulations to illustrate the sharpness of our theoretical predictions.
REF proved that under mild conditions, any stationary point of the non-convex objective function will close to the underlying true parameters.
Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima
ArXiv
ArXiv
Computer Science Mathematics