It is shown that the steepest descent and Newton's method for unconstrained nonconvex optimization under standard assumptions may be both require a number of iterations and function evaluations arbitrarily close to O(ǫ −2 ) to drive the norm of the gradient below ǫ. This shows that the upper bound of O(ǫ −2 ) evaluations known for the steepest descent is tight, and that Newton's method may be as slow as steepest descent in the worst case. The improved evaluation complexity bound of O(ǫ −3/2 ) evaluations known for cubically-regularised Newton methods is also shown to be tight.
REF also showed that the O(1/ √ t) rate was tight for the gradient descent method for an unconstrained objective.
On the Complexity of Steepest Descent, Newton's and Regularized Newton's Methods for Nonconvex Unconstrained Optimization Problems
SIAM Journal on Optimization
SIAM Journal on Optimization
Computer Science Mathematics