S TOCHASTICALLY C ONTROLLED C OMPOSITIONAL G RADIENT FOR C OMPOSITION P ROBLEMS
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider composition problems of the form 1 n (cid:80) ni
=1 F i ( 1 n (cid:80) nj =1 G j ( x )) .
Composition optimization arises in many important machine learning applications: reinforcement learning, variance-aware learning, nonlinear embedding, and many others.
Both gradient descent and stochastic gradient descent are straightforward solution, but both require to compute 1 n (cid:80) nj
=
G j ( x ) in each single iteration, which is inefﬁcient-especially when n is large.
Therefore, with the aim of signiﬁcantly reducing the query complexity of such problems, we designed a stochastically controlled compositional gradient algorithm that incorporates two kinds of variance reduction techniques, and works in both strongly convex and non-convex settings.
The strategy is also accompanied by a mini-batch version of the proposed method that improves query complexity with respect to the size of the mini-batch.
Comprehensive experiments demonstrate the superiority of the proposed method over existing methods.
I NTRODUCTION
In this paper, we study the following composition minimization problem, min
[Equation]
[Equation]
where f : R N → R is differentiable and possibly non-convex, each F i : R M
→ R is a smooth function, each G i : R N → R M is a mapping function, both the numbers of F i ’s and G j ’s are assumed to be n for simplicity We call G ( x ) :
= 1 n (cid:80) nj =1 G j ( x ) the inner function, and F ( w ) :
= 1 n (cid:80) ni
=
F i ( w ) the outer function
Many machine learning problems can be cast as composition problems that include two ﬁnite-sum structures: reinforcement learning (Sutton et al., 1998; Wang et al., 2017; Liu et al., 2016), variance-averse learning (Lian et al., 2017), and nonlinear embedding (Hinton & Roweis, 2003; Dikmen et al., 2015).
In particular,
• ( reinforcement learning )
The S × S system of Bellman equations Wang et al.
(2017) can be written as min x ∈ R S (cid:107) E [ B ] x − E
[ b ] (cid:107) 2 , where E [ B ] =
I − γP π , γ ∈ (0 , 1) is a discount factor, P π is the transition probability under policy π , and E [ b ] is the expected state transition reward.
This is one of key problems in reinforcement learning for evaluating the value of a policy π
. • ( risk-averse learning )
The risk-averse learning Lian et al.
(2017) aims to maximize the expected return while control the variance (or risk) in the meantime:
[Equation]
where
h ( x ; a ) is the loss function including a random variable a , λ > 0 is a regularization parameter.
• ( nonlinear embedding )
Stochastic nonlinear embedding Hinton & Roweis (2003) aims
to map a group of points from a high dimensional space to a low dimensional space by minimizing the KL divergence.
It is a non-convex composition optimization
[Equation]
where p i | t and q i | t are the conditional probabilities w.r.p.
{ z i } n i =1 and { x i } n i =1 , p i | t =
d ( z t ,z i ) (cid:80) j (cid:54) = t d ( z t ,z j ) , q i | t = d ( x t ,x i ) (cid:80) j (cid:54) =
t d ( x t ,x j ) , where d ( · , · ) is the dissimilar distance function between two samples.
To solve the composition optimization including the ﬁnite-sum structure in (1.1), two most straightforward approaches are the gradient descent (GD) and the stochastic gradient descent (SGD).
However, it is extremely expensive to scan all the inner functions (for both SGD and GD) as well as all the outer functions (for GD) in each iteration.
However, note that, unlike solving common stochastic optimization problems, randomly sampling one inner function and one outer function does not give an unbiased estimate for the true gradient; that is, E i ∼ [ n ] ,j ∼ [ n ] [( ∂G j ( x ))
T ∂F i ( ˜ G ( x ))] (cid:54) = ∇ f ( x ) , where ˜ G ( x ) is the estimation of G ( x ) .
The key to solving this composition objective is how to estimate the value of G ( x k ) and its Jacobian with high accuracy using only a few samples in each iteration.
Recently, many stochastic optimization methods solving the composition problem have been developed, such as the stochastic gradient based method and the variance-reduction based method.
For example, stochastic compositional gradient descent (SCGD) (Wang et al., 2017; Liu et al., 2016) estimates the inner function G ( x ) by using an iterative weighted average of the past values of G ( x ) , and then performs the stochastic quasi-gradient iteration.
The advantage of this method is that convergence rate does not depend on n ; however, it queries more samples to the desired point.
Another set of approaches is based on variance reduction – for instance, compositional stochastic variance reduction gradient (Compositional-SVRG)
(Lian et al., 2017) estimates the inner function G ( x ) and the gradient of function f ( x ) by using the variance reduction technique; however, the derived linear convergence rate is related to n .
Motivated by a few recent works (Lei & Jordan, 2017; Lei et al., 2017; Allen-Zhu, 2017) that focus on the stochastically controlled gradient, we were inspired to look for a way to improve the query complexity and reduce the dependence on n to solve the composition optimization in (1.1).
Hence, this paper presents a novel and more efﬁcient method named stochastically controlled compositional gradient (SCCG) for solving composition problems involving a two-ﬁnite-sum structure.
The result is improved query complexity over existing approaches.
Further, all results in this paper can be easily extended to cases where the number of F i and the number of G j are different.
The main contributions of this article are summarized below.
• We provide a stochastically controlled function to estimate the inner function G ( x ) .
Inspired by stochastically controlled stochastic gradient (SCSG) (Lei & Jordan, 2017) that estimates the gradient, G ( x ) can also be estimated by using a snapshot ˜ x s , in which G (˜ x s ) is not computed directly, but is estimated through a random subset from [ n ] .
This is the ﬁrst time that a stochastically controlled function has been incorporated into the process of estimating the inner function.
We have also analyzed how the size of the subset might inﬂuence the query complexity for both strongly convex and non-convex functions.
• We provide a stochastically controlled compositional gradient to estimate the ∇ f ( x ) .
However, there are two potential situations that could be encountered in the estimation process that can impede convergence.
First, the expectation of the gradient is no longer an unbiased estimation; and, second, the gradient of f (˜ x s ) at the snapshot is formed by two random subsets, which are used for the functions F i and G j respectively.
Moreover, the biased gradient bring more difﬁculty in proving the convergence, which are greatly different from those encountered in (Lei & Jordan, 2017; Lian et al., 2017; Lei et al., 2017).
To address these scenarios, we have identiﬁed a bound on the size of the subsets that are used to estimate the gradient.
The details of the analysis can be referred to Section 3.1 and 3.2.
• A mini-batch version of the proposed algorithm is also provided for both strongly convex and non-convex functions.
The corresponding query complexities are improved according to the size of the mini-batch.
More information can be referred to Section 3.3.
1. R ESULTS
Following the classical benchmark for a general problem, the composition algorithm is also to ﬁnd a point x satisfying f ( x ) − f ( x ∗ ) ≤ (cid:15) for a convex function, where x ∗ is the optimal point in the strongly convex function, and (cid:107)∇ f ( x ) (cid:107) 2 ≤ (cid:15) for a non-convex function, respectively. The elegance [Table]
of a composition algorithm is evaluated based on its query complexity, deﬁned as the number of queries in a given sampling oracle that are needed to compute the gradient.
Here, we give the query complexities of the composition problem in Table 1, which offers an insightful comparison to other algorithms.
Strongly convex function The query complexity for the strongly convex function is
O ((min { n, 1 / ( (cid:15)µ 2 ) } + L 2 f /µ 2 min { n, 1 /µ 2 } ) log(1 /(cid:15) )) .
The result is the general form for the strongly convex composition and is equal to or better than the query complexity in (Lian et al., 2017) and (Liu et al., 2017a).
Non-convex function The query complexity is O ( min { 1 /(cid:15) 9 / 5 , n 4 / 5 /(cid:15) } ) , which is better than the result in (Liu et al., 2016) and comparable to the result in (Liu et al., 2017b).
Mini-batch 1 For the mini-batch version, the query complexity can be improved to some extent, that is O ((min { n, 1 / ( (cid:15)µ 2 ) }
+ L 2 f / ( bµ 2 )min { n, 1 /µ 2 } ) log(1 /(cid:15) )) .
and O ( min { 1 /(cid:15) 9 / 5 , n 4 / 5 /(cid:15) } /b 1 / 5 ) for strongly convex and non-convex functions, respectively, which are better than mini-batch variance reduced stochastic compositional proximal gradient method (VRSC-PG)
(Huo et al., 2018) and mini-batch Composite SAGA (C-SAGA) (Yuan et al., 2019) when b= 1 /µ and min { n, 1 /(cid:15) } 2 / 3 for strongly convex and non-convex functions, respectively.
1. R ELATED WORK
As the amount of data we have at our disposal grows, stochastic optimization has become a popular technique in the realm of machine and deep learning, particularly for optimizing ﬁnite-sum functions.
The typical algorithms for solving such problems include stochastic gradient descent (Ghadimi & Lan, 2016), SVRG (Johnson & Zhang, 2013; Reddi et al., 2016), stochastic dual coordinate ascent (SDCA) (Shalev-Shwartz & Zhang, 2014; 2013) and the accelerated Nesterov’s method (Nesterov, 2013), accelerated randomized proximal coordinate (APCG)
(Lin et al., 2014; 2015) and Katyusha method (Allen-Zhu, 2017).
The standard procedure for optimizing a problem with a ﬁnite-sum structure is to randomly select one or a block of components to estimate the gradient.
However, knowing that the estimated gradient usually has a large variance, the gradient of the function is estimated from a snapshot to appropriately reduce the variance – in other words, the procedure includes a variance reduction mechanism.
Composition optimization problems can also be solved with the above algorithms, but the twoﬁnite-sum structures in composition problems mean that when the gradient of the inner function is estimated directly, the query complexity can substantially increase.
Recently, Wang et al.
(2017) proposed a method based on ﬁrst-order SCGD to overcome this issue where the variable and the inner function are updated alternately in two steps.
The method has a query complexity of O ( (cid:15) − 7 / 2 ) for a general function and O ( (cid:15) − 5 / 4 ) for a strongly convex function.
Liu et al.
(2016) employed Nes- terov’s method to accelerate the composition problem, reaching O ( (cid:15) − 5 / 4 ) and O ( (cid:15) − 9 / 4 ) for strongly convex and non-convex functions, respectively.
Ghadimi et al.
(2018) proposed a nested averaged stochastic approximation method to ﬁnd an approximate stationary point within the problem, resulting in a sample complexity of O (1 /(cid:15) 2 ) .
However, these methods estimate the inner function using an iterative weighted average of the past function.
The other stream of solutions focuses on variance reduction technology.
For instance, Lian et al.
(2017) initially applied the SVRG-based method to estimate the inner function G ( x ) and the gradient of the function f ( x ) , which yields a linear convergence rate.
Subsequently, Liu et al. (2017a) and
Devraj & Chen (2019) applied a dual-based method to composition problem, which also yields a linear convergence rate.
Devraj & Chen (2019) also applied the stochastic variance reduced primal-dual algorithms to composition problem.
Yu & Huang (2017) turned to an ADMM-based Boyd (2011) method and provided an analysis of convex functions that do not rely on Lipschitz smoothness.
Moreover, Liu et al.
(2017b) went a step further and considered non-convex functions, analyzing the query complexity with both inner and outer functions of different sizes.
Lin et al.
(2018) considered non-smooth convex composition functions, offering an incremental ﬁrst-order oracle complexity analysis.
Zhang & Xiao (2019) and Huo et al.
(2018) also provided an randomized incremental gradient method for the composition problem including regularization.
Many recent articles have discussed variance reduction methods that estimate the gradient from a random subset rather than through direct computation.
Lei & Jordan (2017), for example, proposed an SCSG method for a convex ﬁnite-sum function.
They then applied it to a non-convex problem in (Lei et al., 2017) by using less than a single pass to compute the gradient at the snapshot point.
Furthermore, Allen-Zhu (2017) proposed the Natasha1.5 algorithm, in which the gradient for each epoch is based on a random subset.
Moreover, the objective function has the regularization term.
Liu et al.
(2018) applied an SCSG based method to the zeroth-order optimization problems with the ﬁnite-sum function.
Recently, Yuan et al.
(2019) applied the stochastic recursive gradient descent method to the composition problem.
The rest of paper is organized as follows: in Section 2, we give preliminaries used for analyzing the proposed algorithm.
Section 3 presents the SCSG-based method for the strongly convex and non-convex composition problem and the corresponding mini-batch version.
In Section 4, we give the experimental results.
We conclude our paper in Section 5.
P RELIMINARIES
Throughout this paper, we use the Euclidean norm denoted by (cid:107) · (cid:107) .
We use i ∈ [ n ] to denote that i is generated from [ n ] = { 1 , 2 , ..., n } .
We denote by ( ∂G ( x ))
T ∇ F ( G ( x ))
the full gradient of the function f , ∂G ( x ) the Jacobian of G , and ( ∂G j ( x ))
T ∇ F i ( G ( x )) as the stochastic gradient of the function f , where i and j are randomly and independently selected from [ n ] .
We use A = |A| to denote the number of elements in the set A , and deﬁne G A ( x ) = 1 A (cid:80) 1 ≤ j ≤ A G A [ j ] ( x ) .
We use
E to denote the expectation, that is E A
[ v ] = 1 A (cid:80) 1 ≤ i ≤ A v A
[ i ] .
Note that all the variables such as subsets A and B , elements i and j are independently selected from [ n ] , in particular, the element in A and B are independent.
So we use E in instead of E i , E j , E A and E B except when explicitly stated otherwise.
Recall deﬁnitions on Lipschitz function, smooth function and strongly convex.
Deﬁnition 1.
For function p on X , ∀ x, y ∈ X , A function p is a B p -Lipschitz, that is (cid:107) p ( x ) − p ( y ) (cid:107) ≤ B p (cid:107) x − y (cid:107) ;
A function p is a L p -smooth, that is (cid:107)∇ p ( x ) − ∇ p ( y ) (cid:107) ≤
L p (cid:107) x − y (cid:107) ; A function p is a µ -strongly convex, that is p ( y ) ≥ p ( x )+ (cid:104) p ( x ) , y − x (cid:105) + µ/ 2 (cid:107) x − y (cid:107) 2 .
Through our discussions, we make the following assumptions,
Assumption 1.
Let B G , L F and L f be positive scalars,
• G j is B G -Lipschitz, j ∈ [ n ] , that is (cid:107) G j ( x ) − G j
( y ) (cid:107) ≤ B G (cid:107) x
− y (cid:107) .
• F i is L F -smooth, i ∈ [ n ] , that is (cid:107)∇ F i ( x ) − ∇ F i ( y ) (cid:107)
≤ L F (cid:107) x − y (cid:107) .
• For function F i ( G ( x )) , there exists a constant L f satisfying (cid:107) ( ∂G j ( x ))
T ∇ F i ( G ( x )) − ( ∂G j ( y ))
T ∇ F i ( G ( y ))
(cid:107) ≤ L f (cid:107) x − y (cid:107) , ∀ i, j ∈ [ n ] .
• We assume that i and j are independently and randomly selected from [ n ] , z ∈ R M , x ∈ R N , then E [( ∂G j ( x ))
T ∇ F i ( z )]
= ( ∂G ( x ))
T ∇ F ( z ) .
Furthermore, we deﬁne H 1 and H 2 are the upper bounds on the variance of G ( x ) and ( ∂G ( x ))
T ∇ F ( y ) , respectively, that is,
[Equation]
In the paper, we denote by x sk the k -th inner iteration at s -th epoch.
But in each epoch analysis, we drop the superscript s and denote by x k for x sk .
We let x ∗ be the optimal solution of the convex f ( x ) .
Throughout the convergence analysis, we use O ( · ) notation to avoid many constants, such as
B G , L F , and L f , that are irrelevant with the convergence rate.
S TOCHASTICALLY C ONTROLLED C OMPOSITIONAL G RADIENT
In this section, we present the variance-reduction based method for the composition problem, which can be used for both the strongly convex function and non-convex function.
Before describing the proposed algorithm, we recall the original SVRG (Johnson & Zhang, 2013).
The general process of SVRG works as follows.
The update process is divided into S epochs, and each of the epoch consists of K iterations.
At the beginning of each epoch, SVRG deﬁnes a snapshot vector ˜ x s , and then compute the full gradient ∇ f (˜ x s ) .
In the inner iteration of the current epoch, SVRG deﬁnes the estimated gradient by randomly selecting i k from [n] at the k -th iteration,
[Equation]
However, for the composition problem, there are also variance-reduction based methods in (Lian et al., 2017; Liu et al., 2017a;b).
The difference with SVRG is that there is another estimated function for G ( x ) , which also has the ﬁnite-sum structure.
These methods deﬁne the estimated function as
[Equation]
where A is the mini-batch formed by randomly sampling from [ n ] .
Whereas, as the number of the inner function G j and the outer function F i increase, it is not reasonable to compute the full gradient of f ( x ) and the full function G ( x ) directly for each epoch.
Extending from the SCSG (Lei et al., 2017; Lei & Jordan, 2017) and Natasha1.5 (Allen-Zhu, 2017), we present a new algorithm SCCG for the composition problem as shown in Algorithm 1.
3 We introduce two subsets D 1 and D 2 , which are independent with each other and randomly selected from [ n ] , respectively.
We deﬁne D
= D 1 ∪D 2 as a new variable, which is important in analyzing the convergence.
Firstly, D 1 is used for estimating the inner function.
Based on the variance reduction technique, the estimated inner function at k -th iteration of s -th epoch is
[Equation]
) where the subset of A is the same as in (3.2).
Note that A and D are independent with each other.
The difference with (3.2) is that the third term in (3.3) is computed under the subset D 1 rather than [ n ] .
Throughout the paper, we assume that |A| ≤ |D 1 | .
Secondly, D 2 is used to estimate the outer function F .
The key distinguish with (Lei et al., 2017; Lei & Jordan, 2017; Allen-Zhu, 2017) is the biased full gradient of f (˜ x s ) .
We deﬁne this estimated full gradient of f (˜ x s ) for each epoch as ∇ ˆ f D (˜ x s ) =
( ∂G D 1 (˜ x s ))
T ∇ F D 2 ( G D 1 (˜ x s )) .
Though E A , D [ ∇ ˆ f D (˜ x s )] (cid:54) = ∇ f
(˜ x s ) , we could still estimate the gradient of the f ( x k ) by
[Equation]
Require: K , S , η , ˜ x 0 and D
= D 1 ∪ D 2 , where D 1 and D 2 are mini-batches.
for s = 0 , 1 , 2 , · · · , S − 1 do
Sample from [ n ] for D times to form mini-batch D
Sample from [ n ] for D times to form mini-batch D∇ ˆ
[Equation] [Equation]
[Equation] [Equation]
)) x
[Equation]
= ˜ x s for k = 0 , 1 , 2 , · · · , K − 1 do
Sample from [ n ] to form mini-batch A
[Equation]
Uniformly and randomly pick i k and j k from [ n ]
Compute the estimated gradient ∇
˜ f k from (3.4) x k +
= x
k − η ∇ ˜
f
k end for
Update ˜ x s +1 = x K , or ˜ x s +
= x r , r is randomly selected from [ K − 1] end for
Output: ˆ x s k is uniformly and randomly chosen from s ∈ { 0 , ..., S − 1 } and k ∈ { 0 , .., K − 1 } .
where i k and j k are randomly selected from [n] at the k -th iteration for functions F and G , respectively.
Furthermore, E i j ,j k A , D [ ∇ ˜ f k ] (cid:54) =
∇ f ( x k ) as well.
This gives us more discussion about the upper bound with respect to the estimated function and the gradient under the new random subset D (more details can be referred to appendix).
3. SCCG FOR THE STRONGLY CONVEX CASE
In this subsection, we analyze the query complexity for the strongly convex composition problem and show that our result is better or comparable to the previous methods.
Furthermore, we discuss the query complexity under different value with respect to n , µ and (cid:15)
Theorem 1.
In Algorithm 1, for the µ -strongly convex problem, suppose Assumption 1 holds, let the step size is η ≤ µ/ (135 L 2 f ) , the subset size of A is A = min { n, 128 B 4 G L 2 F /µ 2 } , the subset size of
D 1 and D 2 are both D = min (cid:8) n, 5 (cid:0) 16 B 4 G L 2 F H
+ 4 H 2 (cid:1) / (4 (cid:15)µ 2 ) (cid:9) , the number of the inner iteration is K ≥ 540 L 2 f /µ 2 , the number of outer iteration is S ≥ 1 / (log(1 /ρ ))
log(
E (cid:107) ˜ x 0 − x ∗ (cid:107) 2 /(cid:15) ) .
Then, the query complexity is ( D + KA ) S
= O (cid:16)(cid:16) min (cid:110) n, 1 (cid:15)µ 2 (cid:111)
+ L 2 f
µ 2 min (cid:110) n, 1 µ 2 (cid:111)(cid:17) log (1 /(cid:15) )
(cid:17) .
As can be seen from the above result, Theorem 1 presents the general query complexity under different parameters ( the details of parameters’ setting can be referred to the Appendix.).
Comparing n with corresponding parameters, we analyze the query complexity separately.
We remove the parameters such as B 2 G , L 2 F , H 1 and H 2 , and analyze the size with the order of 1 /µ 2 .
We consider three internals of n while the min value of the function in the above query complexity will take different results:
• 1 /µ 2 ≤ 1 / ( (cid:15)µ 2 ) ≤ n .
When n is large enough such that we can obtain the query complexity is O ((1 / ( (cid:15)µ 2 ) + L 2 f /µ 4 ) log(1 /(cid:15) )) .
This result avoids the situation that computing the full gradient of f ( x ) and the full function G ( x ) for the large-scale number of n .
What’s more, this result is better than Compositional-SVRG (Lian et al., 2017; Liu et al., 2017a).
• 1 /µ 2 ≤ n ≤ 1 / ( (cid:15)µ 2 ) .
When n is smaller than 1 / ( (cid:15)µ 2 ) , the query complexity becomes O (( n + L 2 f /µ 4 ) log(1 /(cid:15) )) , which is the same as Compositional-SVRG (Lian et al., 2017).
However, we need to compute the full gradient of ∇ f (˜ x s ) as in (3.1).
The estimation of inner function G ( x ) is the same as in (Lian et al., 2017).
• n ≤ 1 /µ 2 ≤ 1 / ( (cid:15)µ 2 ) .
When n is the smallest one, the query complexity becomes O (( n + L 2 f n/µ 2 ) log(1 /(cid:15) )) .
The result has a similar form to SVRG (Johnson & Zhang, 2013).
This also gives us an intuition that the inner function should be computed directly rather than estimated if n is small.
3. SCCG FOR THE N ONCONVEX C ASE
In the previous subsection, we showed convex SCCG converges to the optimal point with improved query complexity.
A natural question is whether the proposed algorithm can improve the performance of the non-convex problem.
We provide an afﬁrmative answer.
In this subsection, we present the query complexity for the non-convex composition problem in term of stationarity gap ||∇ f ( x ) || 2 .
Theorem 2.
In Algorithm 1, for non-convex function, suppose Assumption 1 holds, let the step size is η = min { 1 /n 2 / 5 , (cid:15) 2 / 5 } , the size of the subset D 1 and D 2 are D = min { n, O (1 /(cid:15) ) } , the size of subset A is A = min { n, O (1 /η ) } , the number of inner iteration is K ≤ O (cid:0) 1 /η 3 / 2 (cid:1) , the total number of iteration is T
= O
(1 / ( (cid:15)η )) , in order to obtain E [ (cid:107)∇ f (ˆ x sk ) (cid:107) 2 ] ≤ (cid:15) , the query complexity is O (cid:0) min (cid:8) 1 /(cid:15) 9 / 5 , n 4 / 5 /(cid:15)
(cid:9)(cid:1) .
From the above result, we analyze the query complexity of the non-convex problem separately: 1) when n ≥ 1 /(cid:15) , our query complexity becomes O (1 /(cid:15) 9 / 5 ) , which is independent on n .
This is better than the query complexity of the accelerated method in (Liu et al., 2016), in which the query complexity does not depend on n as well. 2) when n ≤ 1 /(cid:15) , the query complexity becomes
O ( n 4 / 5 /(cid:15) ) , which is consistent with the result of (Liu et al., 2017b) in solving the problem (1.1). [Table]
INI - BATCH VERSION OF SCCG
In this subsection, we present the mini-batch version of the proposed method in Algorithm 2 (in appendix) and obtain the corresponding query complexities for both the strongly convex and the non-convex functions, which provably beneﬁt from mini-batching.
As the process of the proof is similar to that of Theorem 1 and Theorem 2, and the difference with Algorithm 1 is the computation of the gradient of f ( x ) (the corresponding proof of bound is in appendix), we could directly present the corresponding results for both the strongly convex and the non-convex problems.
Corollary 1.
In Algorithm 2, for the µ -strongly convex problem, suppose Assumption 1 holds, let the step size η ≤ bµ/ (135 L 2 f ) , the number of the inner iteration is K ≥ 540 L 2 f / ( bµ 2 ) , in order to obtain
E (cid:107) ˜ x
s − x ∗ (cid:107) 2 ≤ (cid:15) , the query complexity is O (cid:16)(cid:16) min (cid:110) n, 1 (cid:15)µ 2 (cid:111)
+ L 2 f bµ 2 min (cid:110) n, 1 µ 2 (cid:111)(cid:17) log (1 /(cid:15) )
(cid:17) .
Corollary 2.
In Algorithm 2, for the non-convex problem, suppose Assumption 1 holds, let the step size η = b 3 / 5 min { 1 /n 2 / 5 , (cid:15) 2 / 5 } , the number of the inner iteration is K ≤ O (cid:0) b 1 / 2 / ( η 3 / 2 ) (cid:1) , in order to obtain E [ (cid:107)∇ f (ˆ x sk ) (cid:107) 2 ] ≤ (cid:15) , the query complexity is (1 /b 1 / 5 ) O (cid:0) min (cid:8) 1 /(cid:15) 9 / 5 , n 4 / 5 /(cid:15)
(cid:9)(cid:1) .
From the above-given query complexity results for the strongly convex and non-convex problems, we can see that both of their step size η and the number of inner iteration K are larger than the corresponding ones in the non-mini-batch version.
These two key parameters lead to the improved query complexity for both strongly convex and non-convex functions.
E XPERIMENTS
In this section, we evaluate the performance of our proposed algorithm on the strongly convex and non-convex functions, respectively.
SCCG for strongly convex function 4 To verify the effectiveness of the algorithm, we use the mean-variance optimization in portfolio management 5 :
[Equation]
where r i ∈ R N , i ∈ [ n ] is the reward vector, and x ∈ R N is the invested quantity.
In the experimental setting, we set n =3000, |A| = n /10, |D 1 | = 2400 , 2600 , 2800 , which are denoted as SCCG (2400), SCCG (2600) and SCCG (2800).
The reward vectors are generated on Gaussian distribution
[Figure] [Figure]
with the condition number of its covariance matrix denoted by κ cov .
Furthermore, we consider three conditions numbers, κ cov =10, 30 and 50.
We compare our algorithm with the stochastic gradient based methods SCGD and accelerated stochastic method ASC-PG.
Figure 1 shows the performance of the gap between the value function and optimal value, we observe that our algorithm is better than stochastic gradient methods, SCGD and ASC-PG.
SCCG for non-convex function For the non-convex function, we apply the proposed SCCG method to the nonlinear embedding problem in (1.2).
We consider the distance of low-dimension space between x i and x j as 1 / (1 + (cid:107) x i − x j (cid:107) 2 ) , i, j ∈ [ n ] .
Then, the problem can be formulated as the problem in (1.1), in which the details can be referred to the appendix.
We consider three datasets: mnist, Olivetti faces and COIL-20 including different sample sizes and dimensions, 1000 × 784, 400 × 4096 and 1440 × 16384.
Our experiment is to verify our proposed algorithm, thus we set
D
= D 2 in default and choose three different sizes of sample set D 1 , which are smaller than n .
For example, for the case of mnist, we choose |D 1 | = 400 , 600 , 800 , which are denoted as SCCG (400),
SCCG (600) and SCCG (800).
Furthermore, we also set |A| =
n/ 10 , where n is the total number of samples.
Figure 2 shows the norm of the gradient, and Figure 3 (In the appendix) shows the objection value.
We compare our algorithm with the stochastic gradient based method (SCGD and
ASC-PG)
, and observe that our proposed algorithm is better than SCGD and ASC-PG on both the norm of the gradient and objective function.
C ONCLUSION
In this paper, we propose the variance reduction based method for the strongly convex and nonconvex composition problems.
We apply the stochastically controlled stochastic gradient to estimate inner function G ( x ) and the gradient of f ( x ) .
The query complexity of our proposed algorithm is better than or equal to the current methods on both strongly convex and non-convex functions.
Furthermore, we also present the corresponding mini-batch version of the proposed method, in which the query complexities are improved as well.
Experimental results also conﬁrm that our algorithm achieves better query complexity in a real-world problem.
A T ECHNICAL T OOL
For the subset A ⊆ [ n ] , we present the following lemma that the variance of a random variable decreases by a factor |A| if we choose |A| independent elements from [ n ] and average them.
The proof process is trivial However, it is an important tool for analyzing the query complexity under the different sizes of the subsets.
Lemma 1.
If v 1 , ..., v n ∈ R d satisfy (cid:80) ni
=1 v i = (cid:126) 0 , and A is a non-empty, uniform random subset of [ n ] and A = |A| , that is elements in A are uniformly selected from [n] without replacement, then
[Equation]
Furthermore, if elements in A are independently selected from [ n ] with replacement,, then
[Equation]
Proof.
Based on the (cid:80) ni =1 v i = (cid:126) 0 , and permutation and combination,
For the case that A is a non-empty, uniformly random subset of [ n ] , we have
[Equation]
where C nA refer to the number of the combination of n things taken A at a time without repetition.
Thus
[Equation]
we have
[Equation]
For the case that the element in A is randomly and independently selected from [ n ] , we have
[Equation]
Based on Lemma 1, we can obtain the inequality with two-variables D 1 and D 2 , which are used for the gradient of f ( x ) .
Lemma 2.
If w 1 , ..., w n ∈ R M × N and v 1 , ..., v n ∈ R M satisfy ( 1 n (cid:80) i ∈ [ n ] w i ) T ( 1 n (cid:80) j ∈ [ n ] v j ) =
¯ w T ¯ v , and D =
[ D 1 , D 2 ] is a non-empty, uniform random subset consist of D 1 and D 2 , which are independently and uniformly selected from [ n ] , D = |D 1 | =
|D 2 | , then
[Equation]
B B OUND ANALYSIS OF SCCG FOR THE COMPOSITION PROBLEM B. B OUNDS ANALYSIS OF THE ESTIMATED FUNCTION AND THE GRADIENT
Here, we mainly give different kinds of bounds for the proposed algorithm, such as E A , D 1 (cid:107) ˆ G k −
G ( x k ) (cid:107) 2 , E D (cid:107) E A ,i k ,j k [ ∇ ˜ f k ] − ∇ f ( x k ) (cid:107)
and E i k ,j k , A , D
(cid:107)∇ ˜ f k − ∇ f ( x k ) (cid:107) 2 .
These bounds will be used to analyze the convergence rate and query complexity.
These bounds are all based on Assumption 1.
Parameters such as B G , B F , L G , L F and L f in the bound are all from these Assumptions.
We do not deﬁne the exact value of parameters such as h , A and D , which have a great inﬂuence on the convergence but will be clearly deﬁned in the query analysis.
Our proposed bounds are similar to that of (Lian et al., 2017; Liu et al., 2017a;b), but, the difference lies on that there is an extra subset D , which shows an interesting phenomenon.
That is when the subset D is equal to [ n ] , the corresponding bounds are the same as in (Lian et al., 2017;
Liu et al., 2017a;b).
However, it is the independent subset D that gives more general query complexity result for the problem (1.1).
The following bounds are all used for the composition problem for both convex and non-convex problems based on the Lemma 1 and Lemma 2.
For simplicity, we drop the superscript i k , j k , A and D for the expectation with E in the proof.
Lemma 3.
Suppose Assumption 1 holds, for ˆ G k deﬁned in (3.3) with D = |D 1 | and A = |A| , we have
[Equation]
Proof.
By the deﬁnition of ˆ G k in (3.3), we have
[Equation]
where 1 (cid:13) follows from || a 1 + a 2 || 2 ≤ 2 a
+ 2 a 22 ; 2 (cid:13) is based on Lemma 1 and the following inequality: Through adding and subtracting the term G ( x k ) − G (˜ x s )
[Equation]
we have
[Equation]
where 1 (cid:13) follows from || a + b || 2 ≤ 2 a 2 +2 b 2 ; 2 (cid:13) is based on Lemma 1; 3 (cid:13) follows from the bounded function of G and the upper bound of variance of G .
Note that A and x k are independent; and D and ˜ x s are independent.
Lemma 4.
Suppose Assumption 1 holds, for ˆ G k deﬁned in (3.3) and ∇ ˜ f k deﬁned in (3.4) with
D =
[ D 1 , D 2 ] and D = |D 1 | =
|D 2 | , we have
[Equation]
Proof.
Through adding and subtracting the terms of
[Equation]
we have
[Equation]
where 1 (cid:13) follows from || a 1 + a 2 + a 3 + a 4 || 2 ≤ 4 a
+ 4 a 22 + 4 a 23 + 4 a 24 ; 2 (cid:13) is based on the bounded Jacobian of G and the smoothness of F in Assumption 1, and the upper bound of variance in Lemma 2.
3 (cid:13) is based on Lemma 3 and the upper bound of variance of G ( x ) .
Note that A and x k are independent; and D and ˜ x s are independent.
Lemma 5.
Suppose Assumption 1 holds, for ˆ G k deﬁned in (3.3) and ∇ ˜ f k deﬁned in (3.4) with
D =
[ D 1 , D 2 ] and D = |D 1 | =
|D 2 | , we have
[Equation]
Proof.
Through adding and subtracting the term of ( ∂G j ( x k ))
T ∇ F i ( G ( x k )) , (
[Equation] [Equation]
˜ x s ))
T ∇ F D 1 ( G (˜ x s ))
(Note that, D and ˜ x s are independent), we have
[Equation]
[Table] [Table]
[Equation]
where 1 (cid:13) follows from || a 1 + a 2 + a 3 + a 4 + a 5 || 2 ≤ 5 a
+ 5 a 22 + 5 a 23 + 5 a
+ 5 a 25 ; 2 (cid:13) is based on E [ (cid:107)
X
− E
[ X ] (cid:107) 2 ] =
E [ X 2 − (cid:107) E
[ X ] (cid:107) 2 ] ≤ E [ X 2 ] , the smoothness of F i , the bounded Jacobian of
G ( x ) and the smoothness of F in Assumption 1, and the upper bound of the variance.
3 (cid:13) is based on Lemma 3.
As can be seen from the above results directly, when A and D increase, the upper bounds are close to the bounds in (Lian et al., 2017; Liu et al., 2017a;b).
Though there are extra terms with respect to
A and D , they give us another direction for analyzing the convergence rate and query complexity.
The convergence rate not only depends on the convergence sequence, but also the terms including the event function I .
Thus, we can obtain the lower bound range of A and D that is related to (cid:15) .
Furthermore, this lemma can be applied to analyze the convergence rate and query complexity of the convex and non-convex composition problem.
C P ROOF OF SCCG METHOD FOR COMPOSITION PROBLEM C. P ROOF OF SCCG METHOD FOR S TRONGLY CONVEX COMPOSITION PROBLEM
In this section, we analyze the proposed algorithm for the strongly convex composition problem.
We ﬁrst present the convergence of the proposed algorithm and then give the query complexity.
Though the proof is similar to that of (Lian et al., 2017) and (Xiao & Zhang, 2014), we present a more clear and simple process as there is an extra term derived from the subset D .
In order to ensure the convergence of the proposed algorithm, we obtain the desired parameters’ setting, such as A , D , K , η and h .
Based on the setting, we can obtain the corresponding query complexity, which is better than or equal to the SVRG-based method in (Lian et al., 2017) and (Liu et al., 2017a).
This is in fact that the event function I has an inﬂuence on the size of A and D .
C.1. C ONVERGENCE ANALYSIS
Based on the strong convex and smoothness of the function of f ( x ) , we provide the convergence sequence, in which the parameters are not deﬁned.
But the sequences motivate us to consider the parameters’ setting such that lead to the desired convergence rate.
Note that, D and ˜ x s are independent.
Theorem 3.
Suppose Assumption 1 holds, in Algorithm 1, let h > 0 , η > 0 , A = |A| , D = |D 1 | =
|D 2 | , K is the number of the inner iteration, x ∗ is the optimal point, we have
[Equation]
where ρ =
( 1 K + ρ 2 ) /ρ 1 , ρ 2 and ρ 3 deﬁned ( V , V 1 are deﬁned in (C.4) and (C.5).)
[Equation] [Equation]
Proof.
By the update of x k in Algorithm 1, we have
[Equation]
[Table]
[Equation]
where
[Equation]
1 (cid:13) is based on || a 1 + a 2 || 2 ≤ 2 a 21 + 2 a 22 and (cid:104) a 1 , a 2 (cid:105) ≤ h || a 1 ||
+ 1 h
||
a 2 || 2 , h > 0 ; 2 (cid:13) is based on strongly-convex of f in Assumption 1, and Lemma 4, 5.
Summing up from k = 0 to k
= K − 1 , we have
[Equation]
where
[Equation] [Equation]
For x 0 = ˜ x s , by arranging, we have
[Equation]
we assume that ρ 1 > 0 in (C.1), then we can obtain
[Equation]
where ρ =
( 1 K + ρ 2 ) /ρ 1 , ρ 2 and ρ 3 deﬁned in (C.2) and (C.3), the last inequality is based on the formula of geometric progression.
Thus, if ˜ x S converges to the optimal point x ∗ , we need to ensure that ρ < 1 and the second term
ρ 3 (1 − ρ S ) / ( ρ 1 (1 − ρ )) is less than (cid:15)/ 2 .
Actually, if D = n , the second term is equal to zero, which will be similar to the convergence results in (Lian et al., 2017) and (Liu et al., 2017a).
Proof of Theorem
Proof.
In order to keep the proposed algorithm converge, we consider the parameters’ setting, we ﬁrst ensure that ρ 1 > 0 in (C.1), and then deﬁne
[Equation]
that require ρ < 1 , where ρ 2 deﬁned in (C.2).
Thus, the convergence sequence is
[Equation]
We ensure ρ 3 ρ 1 11 − ρ ≤ 12 (cid:15) , where ρ 3 deﬁned in (C.3), that we can derive the size of the D .
In the following we analyze the parameters’ setting such that satisfying the above requirement.
In order to ensure ρ 1 > 0 in (C.1), we consider the parameter h , η and A , (a) h = µ , consider ρ 1 in (C.1), we should require that h ≤ µ , however, V in (C.4) has the relationship with A and D .
In order to keep A small enough, we set the upper bound of h .
Thus, we set h = µ .
(b) A = min (cid:110) n, 128 B 4 G L 2 F 1 µ 2 (cid:111) , based on the setting of h, we require that V/h < µ 16 .
Thus, we have
[Equation]
For V deﬁned in (C.4), if A < n , we have
[Equation]
, otherwise, A = n satisfy the requirement.
Thus, we have A = min (cid:110) n, 128 B 4 G L 2 F 1 µ 2 (cid:111) .
(c) η ≤ 3 µ 53 L 2 f , back to the target of ρ 1 >
, we require that η ≤ 3 µ 53 L 2 f ≤
[Equation] [Equation]
34 µ 12 L 2 f
[Equation]
In order to ensure ρ < 1 in (C.8), we ﬁrst consider ρ 1 and ρ 2 in (C.1) and (C.2).
By the setting of h = µ and V < µ 2 / 16 , we have,
[Equation]
We require that ρ = 1 Kρ 1 + ρ 2 ρ 1 < 1 , and analyze the two terms separately, (a) In order to ρ 2 ρ 1 < 12 , that is
[Equation]
We get η ≤
[Equation]
(b) In order to 1 Kρ 1 < 12 , that is
[Equation]
Thus, we have K ≥
L 2 f µ 2 .
3. Consider the term ρ S E (cid:107) ˜ x 0 − x ∗ (cid:107)
+ ρ 3 ρ 1 11 − ρ , we analyze them separately, (a)
In order to ensure ρ 3 ρ 1 11 − ρ ≤ 12 (cid:15) , that is
[Equation]
Based on the bound of ρ 1 in (C.9)
[Equation]
the deﬁnition of V 1 in (C.5) and the step size η mentioned above
[Equation]
we have
i.
For
V
[Equation]
thus, we have
[Equation]
ii.
If D < n , we can obtain D ≥
[Equation]
, otherwise D = 0 , the above inequality is correct.
Thus, we obtain D = min (cid:110)
[Equation]
[Table]
[Equation]
[Table]
In order to ensure ρ S E
[Table]
x
[Table]
x ∗
[Equation]
2 ≤
[Table]
, we need the number of the outer iterations
[Equation]
All in all, we consider the query complexity based on above parameters’ setting.
For each outer iteration, there will be ( D + KA ) queries.
Thus, the query complexity is
[Equation]
C. P ROOF OF SCCG METHOD FOR N ON - CONVEX COMPOSITION PROBLEM C.2. R ELATED BOUNDS
Lemma 6.
Suppose Assumption 1 hold, in Algorithm 1, we can obtain the following new sequence with respect to f ( x k ) and
|| x k − ˜ x s || 2 , let h > 0 , η > 0 , A = |A| and D =
|D 1 | =
|D 2 | , we have
E [ f ( x k +1 )]
+ c k +1 E (cid:107) x k +1 − ˜ x s (cid:107) 2 ≤ E [ f ( x k )]
+ c k E (cid:107) x k − ˜
x s (cid:107) 2 − u k (cid:107)∇ f ( x k ) (cid:107)
+ J k , where
[Equation]
=
[Equation]
(C.11)
[Equation]
C
[Equation] [Equation]
Proof.
Consider the upper bound of f ( x k +1 ) and (cid:107) x k +1 − ˜ x s (cid:107) 2 , respectively, • Base on the smoothness of f in Assumption 1 and take expectation with respective to i k , j k , we have
[Equation]
[Table]
2 , where the last inequality is based on || a 1 + a 2 || 2 ≤ 2 a 21 + 2 a 22 .
• Base on the update of x k in Algorithm 1 and take expectation with respective to i k , j k , we have,
[Equation]
where the inequality is based on
[Equation]
||
[Equation]
Combine above equalities and Lemma
[Equation] [Equation]
we form a Lyapunov function,
E
[Equation]
− ˜
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where
[Equation]
Based on the above inequality with respect to the sequence E [ f ( x k )]
+ c k E (cid:107) x k − ˜
x s (cid:107) 2 and Algorithm 1, we can obtain the convergence form in which the parameters are not clear deﬁned.
Theorem 4.
In Algorithm 1, suppose Assumption 1 holds, we can obtain the following new sequence with respect to f ( x k ) and
|| x k − ˜ x s || 2 .
K is the number of inner iterations, S is the number of inner iterations, we have
[Equation]
where ˆ x sk is the output point, J 0 and u 0 are deﬁned in (C.15) and (C.13).
Proof.
Based on the update for c k in (C.12), we can see that c k > c k +1 .
As c k is a decreasing sequence, we have u 0 < u k and J k < J 0 .
Then, we get
[Equation]
Sum from k = 0 to k = K − 1 , we can get
[Equation]
Since x 0 = ˜ x s , let ˜ x s +
=
x
K , we obtain,
[Equation]
Summing the outer iteration from s = 0 to S − 1 , we have
[Equation]
where x sk indicates the s -th outer iteration at k -th inner iteration, and ˆ x sk is uniformly and randomly chosen from s = { 0 , ..., S − 1 } and k= { 0 , .., K − 1 } .
C.2. C ONVERGENCE ANALYSIS
Base on Algorithm 1, the analysis of convergence is based on the smoothness of f ( x ) and the update of x under the Lyapunov function to form the convergence sequence.
Theorem 1 shows that our proposed algorithm can converge to the stationary point.
The convergence proof is similar to that of (Liu et al., 2017b; Reddi et al., 2016), however, our algorithm considers the inexact computation of the gradient at the beginning of each epoch.
Thus, we derive the different parameters’ setting.
In particular, the number of the subset D and A depend on the min function.
Intuitively, we can compute the gradient and inner function based on the subset rather on the whole sample.
Moreover, considering the convergence results, we can see that the step size η has the relationship with many parameters, such as the subset A , inner iteration K and the total iteration T .
Proof of Theorem
Proof.
In order to have E [ (cid:107)∇ f (ˆ x s k ) (cid:107) 2 ] ≤ (cid:15) , that is
[Equation]
we consider the corresponding parameters’ setting:
For the ﬁrst term, consider c k deﬁned in (C.12) deﬁne c k =
c
k +1 Y
+ U , for k = K , we have
[Equation]
where
[Equation]
By setting c K → 0 , we obtain
[Equation]
Then, putting the Y and U into the above equation.
We have
[Equation]
where
C =
Y K − 1 .
Because c 0 has the inﬂuence on the parameters such as K , C and u 0 , we analyze them separately,
(a) For K and C , based on the character of function (cid:16) 1 + 1 t 2 (cid:17)
[Equation]
[Equation] e , 6 as t 1 , t 2 → + ∞ and t 1 t 2 < 1 , and the function is also the increasing function with an upper bound of e , we require K < 1 /
[Equation]
.17) thus, we have C <
e − 1 .
(b) For u 0 deﬁned in (C.13), in order to keep u k > 0 , we need to keep c 0 h < 1 / 4 .
If c 0 h < 1 / 4 , there exits a constant ˜ u such that u 0 = ˜ uη .
In order to satisfy c 0 h < 1 / 4 , combine with (C.16) and C <
e − 1 , that is
[Equation]
i. By setting h
=
[Equation]
L 3 f η , there exist ˜ w > 0 , based on above inequality, we have
[Equation]
Thus, combine with the deﬁnition of W in (C.11), we require that
[Equation]
If A < n , we require A ≥ O (cid:16) B 4 G L 2 F / ( L 3 f η ) (cid:17) .
Thus, we have A = min { n, O (1 /η ) } .
ii.
Based on the setting of h and W , combing with (C.17), we have
[Equation]
For the second term about J 0 , as u 0 = w 1 η , we require
[Equation]
Then, if D < n , we require that
[Equation]
Thus, we set D = min { n, O (1 /(cid:15) ) } .
Based on the ﬁrst term L f ( f ( x 0 ) − f ( x ∗ )) ηSK
[Equation]
, the total number of iteration is T =
SK =
[Equation]
Thus, based on the above parameters’ setting, we can ensure that E [ (cid:107)∇ f (ˆ x sk ) (cid:107) 2 ] ≤ (cid:15) .
Based on the parameters’ setting, that is D = min { n, O (1 /(cid:15) ) } , A = min { n, O (1 /η ) } , K ≤
O (cid:0) 1 /η 3 / 2 (cid:1) , and T =
O (1 / ( (cid:15)η )) , we have,
O
[Equation]
where the optimal η = min (cid:8) 1 /n 2 / 5 , (cid:15) 2 / 5 (cid:9)
[Equation]
D P ROOF FOR THE M INI - BATCH OF THE SCCG TO THE COMPOSITION PROBLEM
We provide the Mini-batch version of SCCG:
The following lemma is distinguish with Lemma 5 in which the estimated gradient γ is obtained through b times repeat.
Lemma 7.
Suppose Assumption 1 holds, for ˆ G k deﬁned in (3.3) and Λ deﬁned in Algorithm 2 with
D =
[ D 1 , D 2 ] and D = |D 1 | =
|D 2 | , we have
[Equation]
Proof.
Through adding and subtracting the term of 1 b (cid:80) ( i,j ) ∈
I b ( ∂G j ( x k ))
T ∇ F i ( G ( x k )) , 1 b (cid:80) ( i,j ) ∈
I b ( ∂G i (˜ x s ))
T ∇ F i ( G (˜ x s )) , and ( ∂G (˜ x s ))
T ∇ F ( G (˜ x s )) , ( ∂G D 1 (˜ x s ))
T ∇ F D 1 ( G (˜ x s )) , we have
E (cid:107) Λ − ∇ f ( x k ) (cid:107)
Require: K , S , η (learning rate), ˜ x 0 and D =
[ D 1 , D 2 ] for s = 0 , 1 , 2 , · · · , S − 1 do
Sample from [ n ] for D times to form mini-batch D
Sample from [ n ] for D times to form mini-batch D∇ ˆ f D (˜ x s ) =
( ∂G D 1 (˜ x s ))
T ∇ F D 2 ( G D 1 (˜ x s )) x
0 = ˜ x s for k = 0 , 1 , 2 , · · · , K − 1 do
Sample from [ n ] to form mini-batch A ˆ G k
=
G A ( x k ) − G A (˜ x s ) + G D 1 (˜ x s ) Λ 0 =for t=1,...,b do
Uniformly and randomly pick i k and j k from [ n ]
Compute the estimated gradient ∇
˜ f k from (3.4) Λ t +1 = Λ t + ∇ ˜
f
k end for Λ = Λ b /b x k +
= x
k − η Λ end for
Update ˜ x s +1 = x K , or ˜ x s +
= x r , r is randomly selected from [ K − 1] end for
Output: ˆ x sk is uniformly and randomly chosen from s ∈ { 0 , ..., S − 1 } and k ∈ { 0 , .., K − 1 } .
[Table] [Table]
(cid:88) ( i,j )
[Table]
G
[Equation]
˜ x s
[Equation]
− G D
[Equation]
(˜ x s
[Equation]
D
[Equation]
D
[Equation]
H
[Equation]
, where 1 (cid:13) follows from || a 1 + a 2 + a 3 + a 4 + a 5 || 2 ≤ 5 a
+ 5 a 22 + 5 a 23 + 5 a
+ 5 a 25 , and 2 (cid:13) is based on E [ (cid:107)
X − E [ X ] (cid:107) 2 ] =
E [ X 2 − (cid:107) E
[ X ] (cid:107) 2 ] ≤ E [ X 2 ] and Lemma 1, the smoothness of F i , the bounded Jacobian of G ( x ) and the smoothness of F in Assumption 1, and the upper bound of variance in Lemma 2. 3 (cid:13) is based on Lemma 3.
Proof of Corollary
Proof.
Based on the parameters’ setting, that is D = min { n, O (1 /(cid:15) ) } , A = min { n, O ( b/η ) } ,
K ≤ O (cid:0) b 1 / 2 /η 3 / 2 (cid:1) , and T =
O (1 / ( (cid:15)η )) , we have,
O
[Equation]
min (cid:26)
[Equation]
(cid:27)
[Equation]
[Equation]
η 3 /
[Equation]
η (cid:19)(cid:19) =
O (cid:18)
η 1 /(cid:15)b 1 /(cid:18) min (cid:26) n, 1 (cid:15) (cid:27) + b 3 /η 5 /(cid:19)(cid:19) =(
cid:15)b 1 / 2 O (cid:18) min (cid:26) n
[Equation]
1 (cid:15) (cid:27)
[Equation] [Equation]
/η(cid:19)
[Equation]
[Equation]
[Equation]
O (cid:18) min (cid:26) n 4 /(cid:15)
[Equation]
(cid:15) 9 /(cid:27)(cid:19) , where the optimal η = b 3 / 5 min (cid:8) 1 /n 2 / 5 , (cid:15) 2 / 5 (cid:9) .
E E XPERIMENT E. S TRONGLY CONVEX
To verify the effectiveness of the algorithm, we use the mean-variance optimization in portfolio management 7 : min [Equation] R [Equation]
− n n (cid:88)
i
[Equation]
n n (cid:88)
[Equation]
where r i ∈ R N , i ∈ [ n ] is the reward vector, and x ∈ R N is the invested quantity.
The objective function can be transformed as the composition of two ﬁnite-sum functions in (1.1) with the following forms:
[Equation]
where y 1 ∈ R M and y 2 ∈ R .
Note that the function G j ( x ) =
[ x, (cid:104) r j , x (cid:105) ] , and the corresponding Jacobian is [ I, e ] (cid:62) , where I ∈
R N × N is a unit matrix, and e ∈ R N × 1 is all-ones vector.
It is straightforward to prove that the norm of the Jacobian is bounded, i.e. G j ( x ) is B G -Lipschitz.
We choose such example of the composition problem to verify the efﬁciency of the proposed algorithms, because it has been widely used in related researches (Lian et al., 2017; Wang et al., 2017; Lin et al., 2018).
The source code package will be released as soon as possible to ensure the reproducibility.
E. NON - CONVEX
For the non-convex function, we apply the proposed SCCG method to the nonlinear embedding problem in (1.2).
We consider the distance of low-dimension space between x i and x j as 1 / (1 + (cid:107) x i − x j (cid:107) 2 ) , i, j ∈ [ n ] .
Then, the problem can be formulated as the problem in (1.1).
In particular,
[Equation]
where
[Equation]
[Figure]
Note that, consider the function g
( x ) = 1 1+ x 2 , its gradient is ∇ g ( x ) =
2 x (1+ x 2 ) 2 .
For different value of x , we can see that
[Equation]
Thus, we obtain |∇ g ( x ) | ≤ 2 , which is upper bounded.
Based on this results, we can obtain that the norm of Jacobian is also bounded.
Moreover, in practice, wo do not compute the Jacobian directly as the dimension is large.
The matrix of Jacobian is sparse due to the random subset, which greatly save much space.
We consider three datasets: mnist, Olivetti faces and COIL-20 with different sample sizes and dimensions, 1000 × 784, 400 × 4096 and 1440 × 16384.
Our experiment is to verify our proposed algorithm, thus, we set D
= D 2 in default and choose three different sizes of sample set D 1 , which are smaller than n .
For example, for the case of mnist, we choose |D 1 | = 400 , 600 , 800 , which are denoted as SCCG (400), SCCG (600) and SCCG (800).
Furthermore, we also set |A| =
n/ 10 , where n is the total number of samples.
Figure 2(in the main paper) shows the norm of the gradient, and Figure 3 shows the function value.
We compare our algorithm with stochastic gradient based method (SCGD and ASC-PG) and observe that our proposed algorithm is better than SCGD and
ASC-PG on both the norm of the gradient and objective function.