A Projection-free Algorithm for Constrained Stochastic Multi-level Composition Optimization
Anonymous Author(s)
AffiliationAddress email
Abstract
We propose a projection-free conditional gradient-type algorithm for smoothstochastic multi-level composition optimization, where the objective function is anested composition of T functions and the constraint set is a closed convex set.
Ouralgorithm assumes access to noisy evaluations of the functions and their gradients,through a stochastic first-order oracle satisfying certain standard unbiasedness andsecond-moment assumptions.
We show that the number of calls to the stochastic first-order oracle and the linear-minimization oracle required by the proposedalgorithm, to obtain an ϵ -stationary solution, are of order O T ( ϵ − 2 ) and O T ( ϵ − 3 )respectively, where O T hides constants in T .
Notably, the dependence of thesecomplexity bounds on ϵ and T are separate in the sense that changing one does notimpact the dependence of the bounds on the other.
For the case of T = 1 , we alsoprovide a high-probability convergence result that depends poly-logarithmically onthe inverse confidence level.
Moreover, our algorithm is parameter-free and doesnot require any (increasing) order of mini-batches to converge unlike the commonpractice in the analysis of stochastic conditional gradient-type algorithms.
Introduction
We study projection-free algorithms for solving the following stochastic multi-level compositionoptimization problem
[Equation]
(1) where f i : R d i → R d i − 1 , i = 1 , · · · , T ( d 0 = 1) are continuously differentiable functions, thecomposite function F is bounded below by F ⋆ > −∞ and X
⊂ R d is a closed convex set.
Weassume that the exact function values and derivatives of f i ’s are not available.
In particular, weassume that f i ( y ) =
E ξ i [ G i ( y, ξ i )] for some random variable ξ i .
Our goal is to solve the aboveoptimization problem, given access to noisy evaluations of ∇ f i ’s and f i ’s.
There are two main challenges to address in developing efficient projection-free algorithms forsolving (1).
First, note that denoting the transpose of the Jacobian of f i by ∇ f i , the gradient of theobjective function F ( x ) in (1), is given by ∇ F ( x )
= ∇ f T ( y T ) ∇ f T − 1 ( y T − 1 ) · · · ∇ f 1 ( y 1 ) , wherey i =
f i +1 ◦ · · · ◦ f T ( x ) for 1 ≤ i < T , and y T
= x .
Because of the nested nature of the gradient∇ F ( x ) , obtaining an unbiased gradient estimator in the stochastic first-order setting, with controlledmoments, becomes non-trivial.
Using naive stochastic gradient estimators lead to oracle complexities that depend exponentially on T (in terms of the accuracy parameter).
Next, even when T = 1 in the stochastic setting, projection-free algorithms like the conditional gradient method or its sliding variants invariably require increasing order of mini-batches 1 [28, 40, 24, 39, 52], which make theirpractical implementation infeasible.
In this work, we propose a projection-free conditional gradient-type algorithm that achieves levelindependent oracle complexities (i.e., the dependence of the complexities on the target accuracy isindependent of T ) using only one sample of ( ξ i ) 1 ≤ i ≤ T in each iteration, thereby addressing both ofthe above challenges.
Our algorithm uses moving-average based stochastic estimators of the gradientand function values, also used recently by [19] and [4], along with an inexact conditional gradientmethod used by [3] (which in turn is inspired by the sliding method by [28]).
In order to establishour oracle complexity results, we use a novel merit function based convergence analysis.
To the bestof our knowledge, such an analysis technique is used for the first time in the context of analyzingstochastic conditional gradient-type algorithms.
1.1 Preliminaries and Main Contributions
We now introduce the technical preliminaries required to state and highlight the main contributions ofthis work.
Throughout this work, ∥·∥ denotes the Euclidean norm for vectors and the Frobenius normfor matrices.
We first describe the set of assumptions on the objective functions and the constraint set.
Assumption 1 (Constraint set) .
The set X ⊂ R d is convex and closed with max x,y ∈X ∥ x − y ∥
≤ D X .
Assumption 2 (Smoothness) .
All functions f 1 , . . .
, f T and their derivatives are Lipschitz continuouswith Lipschitz constants L f i and L ∇ f i , respectively.
The above assumptions on the constraint set and the objective function are standard in the literatureon stochastic optimization, and in particular in the analysis of conditional gradient algorithms andmulti-level optimization; see, for example, [28], [50] and [4].
We emphasize here that the abovesmoothness assumptions are made only on the functions ( f i ) 1 ≤ i ≤ T and not on the stochastic functions( G i ) 1 ≤ i ≤ T (which would be a stronger assumption).
Moreover, the Lipschitz continuity of f i ’s areimplied by the Assumption 1 and assuming the functions are continuously differentiable.
However,for sake of illustration, we state both assumptions separately.
In addition to these assumptions, wealso make unbiasedness and bounded-variance assumptions on the stochastic first-order oracle.
Dueto its technical nature, we state the precise details later in Section 3 (see Assumption 3).
We next turn our attention to the convergence criterion that we use in this work to evaluate theperformance of the proposed algorithm.
Gradient-based algorithms iteratively solve sub-problems inthe form of
[Equation]
for some β > 0 , g ∈ R d and x ∈ X .
Denoting the optimal solution of the above problem by
P X ( x, g, β ) and noting its optimality condition, one can easily show that
[Equation]
where N X (¯ x ) is the normal cone to X at ¯ x and B (0 , r ) denotes a ball centered at the origin with radiusr .
Thus, reducing the radius of the ball in the above relation will result in finding an approximate firstorder stationary point of the problem for non-convex constrained minimization problems.
Motivatedby this fact, we can define the gradient mapping at point ¯ x ∈ X as
[Equation]
where Π X ( y ) denotes the Euclidean projection of the vector y onto the set X .
The gradient mappingis a classical measure has been widely used in the literature as a convergence criterion when solvingnonconvex constrained problems [35].
It plays an analogous role of the gradient for constrainedoptimization problems; in fact when the set X ≡ R d the gradient mapping just reduces to ∇ F (¯ x ) .
Itshould be emphasized that while the gradient mapping cannot be computed in the stochastic setting, itstill serves as a measure of convergence.
Our main goal in this work is to find an ϵ -stationary solutionto (1), in the sense described below.
[Table]
[Table]
Definition 1.
A point ¯ x ∈ X generated by an algorithm for solving (1) is called an ϵ -stationary point,if we have E [ ∥G X (¯ x, ∇ F (¯ x ) , β ) ∥ 2 ] ≤ ϵ , where the expectation is taken over all the randomnessinvolved in the problem.
In the literature on conditional gradient methods for the nonconvex setting, the so-called Frank-Wolfegap is also widely used to provide convergence analysis.
The Frank-Wolfe Gap is defined formally as
[Equation]
As pointed out by [3], the gradient mapping criterion and the Frank-Wolfe gap are related to eachother in the following sense.
Proposition 1.
[3] Let g X ( · ) be the Frank-Wolfe gap defined in (4) and G X ( · ) be the gradientmapping defined in (3) .
Then, we have ∥G X ( x, ∇ F ( x ) , β ) ∥ 2 ≤ g X ( x, ∇ F ( x )) , ∀ x ∈ X .
Moreover,under Assumption 1, 2, we have g X ( x, ∇ F ( x )) ≤ (cid:104) (1 /β ) (cid:81)
Ti =
L f i + D X (cid:105)
∥G
X ( x, ∇ F ( x ) , β ) ∥ .
For stochastic conditional gradient-type algorithms, the oracle complexity is measured in termsof number of calls to the Stochastic First-order Oracle (SFO) and the Linear Minimization Oracle(LMO) used to the solve the sub-problems (that are of the form of minimizing a linear function overthe convex feasible set) arising in the algorithm.
In this work, we hence measure the number of callsto SFO and LMO required by the proposed algorithm to obtain an ϵ -stationary solution in the senseof Definition 1.
We now highlight our main contributions :
• We propose a projection-free conditional gradient-type method (Algorithm 1) for solving (1).
InTheorem 2, we show that the SFO and LMO complexities of this algorithm, in order to achieve anϵ -stationary solution in the sense of Definition 1, are of order O ( ϵ − 2 ) and O ( ϵ − 3 ) ,
respectively.•
The above SFO and LMO complexities are in particular level-independent
(i
e., the dependenceof the complexities on the target accuracy is independent of T ).
The proposed algorithm isparameter-free and does not require any mini-batches, making it applicable for the online setting.
• When considering the case of T ≤ 2 , we present a simplified method (Algorithm 3 and 4) toobtain the same oracle complexities
Intriguingly, while this simplified method is still parameterfree for T = 1 , it is not when T = 2 (see Theorem 3 and Remark 3.1).
Furthermore, for the caseof T = 1 , we also establish high-probability bounds (see Theorem 5).
A summary of oracle complexities for stochastic conditional gradient-type algorithms is in Table 1.
1.2 Related Work
Conditional Gradient-Type Method.
The conditional gradient algorithm [15, 29], has had arenewed interest in the machine learning and optimization communities in the past decade; see [33,26, 20, 27, 5, 17] for a partial list of recent works.
Considering the stochastic convex setup, [22, 24] provided expected oracle complexity results for the stochastic conditional gradient algorithm.
Thecomplexities were further improved by a sliding procedure in [28], based on Nesterov’s accelerationmethod.
[40, 52, 24] considered variance reduced stochastic conditional gradient algorithms, andprovided expected oracle complexities in the non-convex setting.
[39] analyzed the sliding algorithmin the non-convex setting and provided results for the gradient mapping criterion.
All of the aboveworks require increasing orders of mini-batches to obtain their oracle complexity results .[34] and [54] proposed a stochastic conditional gradient-type algorithm with moving-averagegradient estimator for the convex and non-convex setting that uses only one-sample in each iteration.
However, [34] and [54] require several restrictive assumptions, which we explain next (focusingon [54] which considers the nonconvex case).
Specifically, [54] requires that the stochastic function
G 1 ( x, ξ 1 ) has uniformly bounded function value, gradient-norm, and Hessian spectral-norm, and thedistribution of the random vector ξ 1 has an absolutely continuous density p such that the norm of thegradient of log p and spectral norm of the Hessian of log p has finite fourth and second-momentsrespectively.
In contrasts, we do not require such stringent assumptions.
Next, all of the aboveworks focus only on the case of T = 1 .
[1] considered stochastic conditional gradient algorithm forsolving (1), with T = 2 .
However, [1] also makes stringent assumptions: (i) the stochastic objectivefunctions G 1 ( x, ξ 1 ) and G 2 ( x, ξ 1 ) themselves have Lipschitz gradients almost surely and (ii) for agiven instance of random vectors ξ 1 and ξ 2 , one could query the oracle at the current and previousiterations, which makes the algorithm not to be truly online.
See Table 1 for a summary.
Stochastic Multi-level Composition Optimization.
Compositional optimization problems of theform in (1) have been considered as early as 1970s by [12].
Recently, there has been a renewedinterest on this problem.
[13] and [10] considered a sample-average approximation approach forsolving (1) and established several asymptotic results.
For the case of T = 2 , [48], [49] and [6]proposed and analyzed stochastic gradient descent-type algorithms in the smooth setting.
[9] and[11] considered the non-smooth setting and established oracle complexity results.
Furthermore, [25]proposed algorithms when the randomness between the two levels are not necessarily independent.
For the general case of T ≥ 1 , [50] proposed stochastic gradient descent-type algorithms andestablished oracle complexities established that depend exponentially on T and are hence sub-optimal.
Indeed, large deviation and Central Limit Theorem results established in [13, 10], respectively, showthat in the sample-average approximation setting, the arg min of the problem in (1) based on nsamples, converges at a level-independent rate (i.e., dependence of the convergence rate on the targetaccuracy is independent of T ) to the true minimizer, under suitable regularity conditions.[19] proposed a single time-scale Nested Averaged Stochastic Approximation (NASA) algorithmand established optimal rates for the cases of T = 1 , 2 .
For the general case of T ≥ 1 , [4]proposed a linearized NASA algorithm and established level-independent and optimal convergencerates.
Concurrently, [43] considered the case when the function f
i are non-smooth and establishedasymptotic convergence results.
[53] also established non-asymptotic level-independent oraclecomplexities, however, under stronger assumptions than that in [4].
Firstly, they assumed that fora fixed batch of samples, one could query the oracle on different points, which is not suited forthe general online stochastic optimization setup.
Next, they assume a much stronger mean-square
Lipschitz smoothness assumption on the individual functions f i and their gradients.
Finally, theyrequired mini-batches sizes that depend exponentially on T , which makes their method impractical.
Concurrent to [4], level-independent rates were also obtained for unconstrained problems by [7],albeit, under the stronger assumption that the stochastic functions G i ( x, ξ i ) are Lipschitz, almostsurely.
It is also worth mentioning that while some of the above papers considered constrainedproblems, the algorithms proposed and analyzed in the above works are not projection-free, which isthe main focus of this work.
Methodology
In this section, we present our projection-free algorithm for solving problem (1).
The methodgenerates three random sequences, namely, approximate solutions { x k } , average gradients { z k } ,and average function values { u k } , defined on a certain probability space (Ω , F , P ) .
We let F kto be the σ -algebra generated by { x 0 , . . .
, x k , z 0 , . . .
, z k , u 01 , . . .
, u k 1 , . . .
, u 0 T , . . .
, u kT } .
The overallmethod is given in Algorithm 1.
In (7), the stochastic Jacobians J k +1 i ∈
R
d i × d i − 1 , and the product(cid:81)
Ti =1 J k +1 T
+1 − i is calculated as J k +1 T
[Equation]
∈ R d T × d 1 ≡ R d T × 1 .
In (8), we use the notation
Input: x 0 ∈ X , z 0 =
[Equation]
R
[Equation]
, u
[Equation]
R
[Equation]
, i
[Equation]
, T , β k > 0 , t k > 0 , τ k ∈ (0 , 1] , δ ≥ 0 .
for
[Equation]
i at ki
[Equation]
8) end for
Output: ( x R , z R , u R 1 , · · · , u RT ) , where is uniformly distributed over {
. , N }
Input: ( x, z, β, M, δ )
Set w 0 = x . for t = 0 , 1 , 2 , . . .
, M do
Find v t ∈ X with a quantity δ ≥ 0 such that
[Equation]
Set w t +
=
(1 − µ t ) w t + µ t v t with µ t = min (cid:26) 1 , ⟨ β ( x − w t ) − z,v t −
w t ⟩ β ∥ v t − w t ∥(cid:27) .
end for
Output: w M ⟨· , ·⟩ to represent both matrix-vector multiplication and vector-vector inner product.
There are twoaspects of the algorithm that we highlight specifically: ( i )
In addition to estimating the gradient of
F , we also estimate a stochastic linear approximation of the inner functions
f i by a moving-averagetechnique.
In the multi-level setting we consider, it helps us to avoid the accumulation of bias, whenestimating the f i directly.
Linearization techniques were used in the stochastic optimization sincethe work of [42].
A similar approach was used in [4] in the context of projected-based methods forsolving (1).
It is also worth mentioning that other linearization techniques have been used in [9]and [11] for estimating the stochastic inner function values for weakly convex two-level compositionproblems, and ( ii )
The ICG method given in Algorithm 2 is essentially applying deterministicconditional gradient method with the exact line search for solving the quadratic minimizationsubproblem in (2) with the estimated gradient z k in (7).
It was also used in [3] as a sub-routine and ismotivated by the sliding approach of [28].
Main Results
In this section, we present our main result on the oracle complexity of Algorithm 1.
Before weproceed, we present our assumptions on the stochastic first-order oracle.
Assumption 3 (Stochastic First-Order Oracle) .
Denote u k T +1 ≡ x k .
For each k , u k
i +1 being the input,the stochastic oracle outputs G k +
i ∈
R
d i and J k
+1 i such that given F k and for any i ∈ { 1 , . . .
, T }
[Equation] [Equation]
(c) The outputs of the stochastic oracle at level i ,
G k +1 i and J
k
+
i , are independent.
The outputs of
the stochastic oracle are independent between levels, i.e., {
G k +1 i } i
=1 ,...,T are independent and
so are {
J k +1 i } i =1 ,...,T .
Parts (a) and (b) in Assumption 3 are standard unbiasedness and bounded variance assumptions onthe stochastic gradient, common in the literature.
Part (c) is essential to establish the convergenceresults in the multi-level case.
Similar assumptions have been made, for example, in [50] and [4].
We also emphasize that unlike some prior works (see e.g., [54]), Assumption 3 allows the case ofendogenous uncertainty, and we do not require the distribution of the random variables ( ξ i ) 1 ≤ i ≤ T tobe independent of the distribution of the decision variables ( u i ) 1 ≤ i ≤ T .
Remark.
Under Assumption 2, and 3, we can immediately conclude that E [ ∥ J k +1 i ∥ 2 | F k ] =
E [ ∥ J k +
i
[Equation]
In the sequel, ˆ σ 2 J i will beused to simplify the presentation.
We start with the merit function used in this work and its connection to the gradient mapping criterion.
Our proof leverages the following merit function:
[Equation]
where α, { γ i } 1 ≤ i ≤ T are positive constants and
[Equation]
Compared to [4], we require the additional term ∥∇ F ( x ) − z ∥ 2 , which turns out to be essential inour proof due to the ICG routine.
The following proposition relates the merit function above to thegradient mapping.
Proposition 2.
Let G X ( · ) be the gradient mapping defined in (3) and η ( · , · ) be defined in (10) .
Forany pair of ( x, z ) and β > 0 , we have ∥G X ( x, ∇ F ( x ) , β ) ∥ 2 ≤ − 4 βη ( x, z ) + 2 ∥∇ F ( x ) − z ∥ 2 .
Proof.
By expanding the square, and using the properties of projection operation, we have
[Equation]
Thus, we have η ( x, z ) ≤ − β 2 ∥ Π X ( x − 1 β z ) − x ∥ 2 .
The proof is completed immediately by notingthat ∥G ( x, ∇ F ( x ) , β ) ∥ 2 ≤ 2 β 2 ∥
Π X ( x − 1 β z ) − x ∥ 2 + 2 ∥∇ F ( x ) − z ∥ 2 .
We now present out main result on the oracle complexity of Algorithm1.
Theorem 2.
Under Assumption 1, 2, 3, let { x k , z k , { u ki } 1 ≤ i ≤ T } k ≥ 0 be the sequence generated by
Algorithm 1 with N ≥ 1 and
[Equation]
where β is an arbitrary positive constant.
Provided that the merit function W α,γ ( x, z, u ) is definedas (9) with
[Equation]
we have,
[Equation]
E
[Equation]
β √ N , 1 ≤
[Equation]
14) where u T +1 = x, B ( β, σ 2 , L, D X , T, δ ) = 4ˆ σ 2 +32 βD 2 X (1+ δ ) (cid:16) 35 + 5 L 2 ∇ F β 2 (cid:17) , and ˆ σ 2 is a constantdepending on the parameters ( β, σ 2 , L, D X , T ) given in (42) .
The expectation is taken with respectto all random sequences generated by the method and an independent random integer number Runiformly distributed over { 1 , . . .
, N } .
That is to say, the number of calls to SFO and LMO to get anϵ -stationary point is upper bounded by O T ( ϵ − 2 ) , O T ( ϵ − 3 ) respectively.
Remark.
The constant B ( β, σ 2 , L, D X , T, δ ) is O ( T ) given the definition of ˆ σ 2 and the value of γ jin (12) , which further implies that the total number of calls to SFO and LMO of Algorithm 1 for findingan ϵ -stationary point of (1) , are bounded by O ( T 2 ϵ − 2 )
=
O T ( ϵ − 2 ) and O ( T 3 ϵ − 3 ) =
O T ( ϵ − 3 )respectively.
Furthermore, it is worth noting that this complexity bound for Algorithm 1 is obtainedwithout any dependence of the parameter β k on Lipschitz constants due to the choice of arbitrarypositive constant β in (11) , and τ k , t k depend only on the number of iterations N and k respectively.
This makes Algorithm 1 parameter-free and easy to implement.
Remark.
As discussed in Section 2, the ICG routine given in Algorithm 2 is a deterministic methodwith the estimated gradient z k in (7) .
The number of iterations, t k , required to run Algorithm 2is given by t k = ⌈ √ k ⌉ .
That is, we require more precise solutions for the ICG routine, only forlater outer iterations.
Furthermore, due to the deterministic nature of the ICG routine, furtheradvances in the analysis of deterministic conditional gradient methods under additional assumptionson the constraint set X (see, for example, [16, 18]) could be leveraged to improve the overall LMOcomplexity.
3.1 The special cases of T = 1 and T = 2
We now discuss several intriguing points regarding the choice of tuning parameter β , for the caseof T = 2 , and the more standard case of T = 1 .
Specifically, the linearization technique usedin Algorithm 1 turns out to be not necessary for the case of T = 2 and T = 1 to obtain similarrates.
However, without linearization, the choice of β is dependent on the problem parameters for
T = 2 .
Whereas it turns out to be independent of the problem parameters (similar to Algorithm 1 and
Theorem 2 which holds for all T ≥ 1 ) for T = 1 .
As the outer function value estimates (i.e., u k +11sequence) are not required for the convergence analysis, we remove them in Algorithms 3 and 4.
Algorithm 3 NASA with Inexact Conditional Gradient Method ( NASA+ICG ) for T =
Replace Step 2 of Algorithm 1 with the following:
Update the average gradient z and the function value estimate u 2 respectively as:
[Equation]
and [Equation] Algorithm 4 ASA with Inexact Conditional Gradient Method ( ASA+ICG ) for T =
Replace Step 2 of Algorithm 1 with the following:
Update the average gradient z as: z k +1 =
(1 − τ k )
z k + τ k J k +
Theorem 3.
Let Assumptions 1, 2, 3 be satisfied by the optimization problem (1) .
Let C 1 , C 2 and C 3be some constants depending on the parameters ( β, σ 2 , L, D X , δ ) , as defined in (54) and (62) .
Letτ
= 1 , t 0 = 0 ,
τ k = 1 √ N , t k = ⌈√
k ⌉ , ∀ k ≥ 1 , where N is the total number of iterations.(a)
Let T = 2 , and let { x k , z k , u k 2 } k ≥ [-
-]0 be the sequence generated by Algorithm 3 with
[Equation]
Then, we have ∀ N ≥ 1 ,
[Equation]
(b) Let T = 1 and let { x k , z k } k ≥ 0 be the sequence generated by Algorithm 4 with β k ≡ β >
.Then, we have ∀ N ≥ 1 ,
[Equation]
All expectations are taken with respect to all random sequences generated by the respective algorithmsand an independent random integer number R uniformly distributed over { 1 , . . .
, N } .
In both cases,the number of calls to SFO and LMO to get an ϵ -stationary point is upper bounded by O ( ϵ − 2 ) , O ( ϵ − 3 )respectively.
Remark.
While we can obtain the same complexities without using the linear approximation of theinner function for T = 2 , it seems necessary to have a parameter-free algorithm as the choice ofβ in (15) depends on the knowledge of the problem parameters.
Indeed, the linearization term in(8) helps use to better exploit the Lipschitz smoothness of the gradients get an error bound in theorder of τ 2 k
∥ d k ∥ 2 for estimating the inner function values.
Without this term, we are only able to usethe Lipschitz continuity of the inner functions and so the error estimate will increase to the order ofτ k
∥
d k ∥ .
Hence, we need to choose a larger beta (as in (15) ) to reduce ∥ d k ∥ and handle the errorterm without compromising the complexities.
However, this is not the case for T = 1 as it can beseen as a two-level problem whose inner function is exactly known (the identity map).
In this case,the choice of β is independent of the problem parameters with or without the linearization term.
3.2 High-Probability Convergence for T = 1
In this subsection, we establish an oracle complexity result with high-probability for the case of
T = 1 .
We first provide a notion of ( ϵ, δ ) -stationary point and a related tail assumption on thestochastic first-order oracle below.
Definition 4.
A point ¯ x ∈ X generated by an algorithm for solving (1) is called an ( ϵ, δ ) -stationarypoint, if we have ∥G X (¯ x, ∇ F (¯ x ) , β ) ∥ 2 ≤ ϵ with probability 1 − δ .
Assumption 4.
Let ∆ k +1 =
∇ F ( x k ) − J k + for k ≥ 0 .
For each k , given F k we have
E [∆ k +1 | F k ] = 0 and ∥ ∆ k +1 ∥ (cid:12)(cid:12)
F k is K -sub-Gaussian.
The above assumption is commonly used in the literature; see [23, 21, 30, 55].
We also refer to[45] and Appendix E for additional details.
The high-probability bound for solving non-convexconstrained problems by Algorithm 4 is given below.
Theorem 5.
Let Assumptions 1, 2, 4 be satisfied by the optimization problem (1) with T
= 1 .
Letτ
= 1 , t 0 = 0 ,
τ k = 1 √ N , t k = ⌈√
k ⌉ , ∀ k ≥ 1 , where N is the total number of iterations.
Let T =
1and let { x k , z k } k ≥ 0 be the sequence generated by Algorithm 4 with β k ≡ β > 0 .
Then, we have∀ N ≥ 1 , δ > 0 , with probability at least 1 − δ ,
[Equation]
Therefore, the number of calls to SFO and LMO to get an ( ϵ, δ ) -stationary point is upper bounded by
O ( ϵ − 2 log 2 (1 /δ )) , O ( ϵ − 3 log 3 (1 /δ )) respectively.
Remark.
To the best of our knowledge, the above result is (i) the first high-probability bound forone-sample stochastic conditional gradient-type algorithm for the case of T = 1 , and (ii) the firsthigh-probability bound for constrained stochastic optimization algorithms in the non-convex setting;see Appendix J of [32].
Proof Sketch of Main Results
In this section, we only present the proof sketch.
The complete proofs are provided in the appendix.
For convenience, let u T +
= x , and we denote H k as the function value of the subproblem at step k ,y k as the optimal solution of the subproblem i.e.,
[Equation]
Then, the proof of Theorem 2 proceeds via the following steps:
We first leverage the merit function W k :=
W α,γ ( x k , z k , u k ) defined in (9) with appropriatechoices of α, γ for any β > 0 to obtain
[Equation]
where R k is the residual term (see (31)) and E [ R k | F k ] ≤ ˆ σ 2 τ 2 k , as shown in Proposition 3.2.
Telescoping the above inequality, in Lemma 11 we obtain the following:
[Equation]
To further control the error term H k (˜ y k ) − H k ( y k ) introduced by the ICG method, we set t k , thenumber of iterations in ICG method at step k , to ⌈√ k ⌉ .
By Lemma 8, we therefore have
[Equation]
Also, with the choice of τ k = 1 √ N and z 0 = 0 , we can conclude that
[Equation]
Then, taking expectation of both sides and by the definition of random integer R , we have
[Equation]
∀ N ≥ 1 , where B is a constant depending on the problem parameters ( β, σ 2 , L, D X , T, δ ) .5.
As a result, we can obtain (13) and (14) by noting that ∀ k ≥
[Equation]
where the second inequality follows the non-expansiveness of the projection operator.
The proofs of Theorems 3 and 5 follow the same argument with appropriate modifications.
Thehigh-probability convergence proof of Theorem 5 mainly consists of controlling the tail probabilityof the residual term R k being large.
Discussion
In this work, we propose and analyze projection-free conditional gradient-type algorithms for constrained stochastic multi-level composition optimization of the form in (1).
We show that theoracle complexity of the proposed algorithms is level-independent in terms of the target accuracy.
Furthermore, our algorithm does not require any increasing order of mini-batches under standardunbiasedness and bounded second-moment assumptions on the stochastic first-order oracle, and isparameter-free.
Some open questions for future research: ( i ) Considering the one-sample setting,either improving the LMO complexity from O ( ϵ − 3 ) to O ( ϵ − 2 ) for general closed convex constraintsets or establishing lower bounds showing that O ( ϵ − 3 ) is necessary while keeping the SFO in theorder of O ( ϵ − 2 ) , is extremely interesting; and ( ii ) Providing high-probability bounds for stochasticmulti-level composition problems ( T > 1 ) and under sub-Gaussian or heavy-tail assumptions (as in[32, 31]) is interesting to explore.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] This work considers a particularprojection-free algorithm for a specific class of optimization problems.
See alsoSection 5.(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
This is atheoretical paper which does not have any direct negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assumption1, 2, 3, 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] We included a.ipynb file to reproduce the experimental results.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/ A
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?x [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]