We develop a line-search second-order algorithmic framework for optimization problems in noisy environments, i. e., assuming that only noisy values are available for the objective function and its gradient and Hessian. In the general noisy case, almost sure convergence of the methods fitting into the framework is proved when line searches and suitably decaying step lengths are combined. When the objective function is a finite sum, such as in machine learning applications, our framework is specialized as a stochastic L-BFGS method with line search only, with almost sure convergence to the solution. In this case, linear convergence rate of the expected function error is also proved , along with a worst-case \mathcal{O (\log(\varepsilon^{-1})) complexity bound} .  Numerical experiments, including comparisons with state-of-the art first- and second-order stochastic optimization methods, show the efficiency of our approach.
We develop a line-search second-order algorithmic framework for minimizing finite sums. We do not make any convexity assumptions, but require the terms of the sum to be continuously differentiable and have Lipschitz-continuous gradients. The methods fitting into this framework combine line searches and suitably decaying step lengths (\log(\varepsilon^{-1})) complexity bound} . A key issue is a two-step sampling at each iteration, which allows us to control the error present in the line-search procedure. Stationarity of limit points is proved in the almost-sure sense, while almost-sure convergence of the sequence of approximations to the solution holds with the additional hypothesis that the functions are strongly convex. Numerical experiments, including comparisons with state-of-the art stochastic optimization methods, show the efficiency of our approach.
R
optimization problems in noisy environments, i. e., assuming that only noisy values are available for the objective function and its gradient and Hessian. In the general noisy case, almost sure convergence of the
minimizing finite sums. We do not make any convexity assumptions, but require the terms of the sum to be continuously differentiable and have Lipschitz-continuous gradients. The
R
the framework is proved when
this framework combine
D
are combined. When the objective function is a finite sum, such as in machine learning applications, our framework is specialized as a stochastic L-BFGS method with line search only, with almost sure convergence to the solution. In this case, linear convergence rate of the expected function error is also proved , along with a worst-case \mathcal{O
A
A key issue is a two-step sampling at each iteration, which allows us to control the error present in the line-search procedure. Stationarity of limit points is proved in the almost-sure sense, while almost-sure convergence of the sequence of approximations to the solution holds with the additional hypothesis that the functions are strongly convex.
D
first- and second-order