The stochastic composition optimization proposed recently by Wang et al. [2014] minimizes the objective with the compositional expectation form: It summarizes many important applications in machine learning, statistics, and finance. In this paper, we consider the finite-sum scenario for composition optimization: We propose two algorithms to solve this problem by combining the stochastic compositional gradient descent (SCGD) and the stochastic variance reduced gradient (SVRG) technique. A constant linear convergence rate is proved for strongly convex optimization, which substantially improves the sublinear rate O(K −0.8 ) of the best known algorithm.
Recently, REF further improves the convergence rate to O(ρ K/(m+n+κ 2 ) ) for the finite sum problem (5) by utilizing a stochastic variance reduced gradient algorithm (SVRG).
Finite-sum Composition Optimization via Variance Reduced Gradient Descent
AISTATS 2017
arXiv: Optimization and Control
Computer Science Mathematics