Online Frank-Wolfe with Arbitrary Delays
Abstract
The online Frank-Wolfe (OFW) method has gained much popularity for online convex optimization due to its projection-free property.
Previous studies show that OFW can attain an O ( T 3 / 4 ) regret bound for convex losses and an O ( T 2 / 3 ) regret bound for strongly convex losses.
However, they assume that each gradient queried by OFW is revealed immediately, which may not hold in practice and limits the application of OFW.
To address this limitation, we propose a delayed variant of OFW, which allows gradients to be delayed by arbitrary rounds.
The main idea is to perform an update similar to OFW after receiving any delayed gradient, and play the latest decision for each round.
Despite its simplicity, we prove that our delayed variant of OFW is able to achieve an O ( T 3 / 4 + dT 1 / 4 ) regret bound for convex losses and an O ( T 2 / 3 + d log T ) regret bound for strongly convex losses, where d is the maximum delay.
This is quite surprising since under a relatively large amount of delay (e.g., d = O ( √ T ) for convex losses and d = O ( T 2 / 3 / log T ) for strongly convex losses), the delayed variant of OFW enjoys the same regret bound as that of the original OFW.
Introduction
Online convex optimization (OCO) has become a leading paradigm for online learning due to its capability to model various problems from diverse domains such as online routing, online collaborative ﬁltering, and online advertisement [Hazan, 2016].
In general, it is formulated as a structured repeated game between a player and an adversary.
In each round t , the player ﬁrst chooses a decision x t from a convex decision set K ⊆ R n , where n is the dimensionality.
Then, the adversary selects a convex function f t ( x ) : K (cid:55)→ R , and the player suffers a loss f t ( x t ) .
The player aims to choose decisions such that the regret
[Equation]
is sublinear in the number of total rounds T .
Online gradient descent (OGD) is a standard method for
OCO, which enjoys an O ( √ T ) regret bound for convex losses [Zinkevich, 2003] and an O (log T ) regret bound for strongly convex losses [Hazan et al., 2007].
However, it needs to compute a projection onto the decision set to ensure the feasibility of each decision, which is computationally expensive for complex decision sets [Hazan and Kale, 2012].
To tackle this computational issue, Hazan and Kale [2012] propose the online Frank-Wolfe (OFW) method, which has become one of the most commonly used algorithms for OCO over complex decision sets.
The main advantage of OFW is its projection-free property: instead of performing the projection operation, it utilizes a linear optimization step to select a feasible decision, which could be much more efﬁcient.
For example, in the problem of online collaborative ﬁltering, the decision set consists of matrices with a bounded trace norm, and the linear optimization step is at least an order of magnitude faster than the projection operation [Hazan and Kale, 2012].
Moreover, it has been shown that OFW achieves an O ( T 3 / 4 ) regret bound for convex losses [Hazan and Kale, 2012, Hazan, 2016] and an O ( T 2 / 3 ) regret bound for strongly convex losses [Wan and Zhang, 2021, Garber and
Kretzu, 2021], which are the best known regret bounds of projection-free methods without further assumptions.
However, OFW requires that the gradient ∇ f t ( x t ) is revealed immediately after making the decision x t , which is not necessarily satisﬁed in reality.
For example, in the previously mentioned online collaborative ﬁltering [Hazan and Kale, 2012], the decision is a prediction of a user-item rating matrix, and the corresponding gradient depends on the true rating of a user on a item, which may not be decided by the user immediately.
Therefore, it is natural to consider a more practical setting, where the gradient ∇ f t ( x t ) arrives at the end of round t + d t − 1 , and d t ≥ 1 denotes an arbitrary delay.
To handle this setting, one potential way is to combine OFW with an existing black-box technique for converting any traditional OCO algorithm into this delayed setting [Joulani et al., 2013].
To be precise, this black-box technique is to pool independent instances of OFW, each of which acts as a learner in the non-delayed setting over a subsequence of rounds.
In each round, a single instance will be taken out from the pool, which makes a decision and then waits for its feedback before rejoining the pool.
If the pool is empty, a new instance of OFW will be added to it.
Moreover, according to Joulani et al.
[2013], their black-box technique is able to attain a regret bound of dR ( T/d ) by combining with a traditional OCO algorithm with R ( T ) regret, where d is the maximum delay.
As a result, combining it with OFW will attain an O ( d 1 / 4 T 3 / 4 ) regret bound for convex losses and an
O ( d 1 / 3 T 2 / 3 ) regret bound for strongly convex losses, which magnify the regret bounds of OFW in the non-delayed setting by a coefﬁcient depending the delay.
Thus, it is natural to ask whether the effect of delay can be further reduced.
In this paper, we give an afﬁrmative answer by developing a simple method called delayed OFW, which is robust to a relatively large amount of delay for both convex and strongly convex losses.
Different from the black-box technique that needs to maintain multiple instances of OFW [Joulani et al., 2013], our delayed OFW is a natural extension of OFW in the delayed setting, which updates the decision similar to OFW after receiving any delayed gradient, and plays the latest decision for each round.
Our theoretical contributions are summarized as follows.
• First, we prove that our delayed OFW attains an O ( T 3 / 4 + dT 1 / 4 ) regret bound for convex losses, where d is the maximum delay, which matches the O ( T 3 / 4 ) regret bound in the non-delayed setting as long as d does not exceed O ( √ T ) .
Second, we prove that our delayed OFW attains an O ( T 2 / 3 + d log T ) regret bound for strongly convex losses, which matches the O ( T 2 / 3 ) regret bound in the non-delayed setting as long as d does not exceed O ( T 2 / 3 / log T ) .
Therefore, our regret bounds are strictly better than those achieved by combining the black-box technique [Joulani et al., 2013] with OFW, when the term involving d in them is not dominant.
Furthermore, simulation experiments are conducted to verify the performance of our delayed OFW.
Related work
In this section, we brieﬂy review related work on projection-free algorithms for OCO, and OCO under delayed feedback.
2.1 Projection-free algorithms for OCO
The OFW method [Hazan and Kale, 2012, Hazan, 2016] is the ﬁrst projection-free algorithm for
OCO, which is an online extension of the classical Frank-Wolfe method [Frank and Wolfe, 1956,
Jaggi, 2013].
For convex losses, OFW ﬁrst chooses an arbitrary x 1 ∈ K , and then iteratively updates its decision by the following linear optimization step
[Equation]
where F t ( x ) is a surrogate loss function deﬁned as
[Equation]
and η, σ t are two parameters.
By setting parameters appropriately, it can attain an O ( T 3 / 4 ) regret bound for convex losses.
If losses are convex and smooth, Hazan and Minasyan [2020] propose a randomized projection-free method, which is based on a classical OCO method called follow the perturbed leader [Kalai and
Vempala, 2005], and achieve a regret bound of O
( T 2 / 3 ) .
Recently, Wan and Zhang [2021] prove that OFW can achieve an O ( T 2 / 3 ) regret bound for strongly convex losses.
Speciﬁcally, to utilize the strong convexity of losses, they redeﬁne F t ( x ) in (2) to
[Equation]
where β is the modulus of the strong convexity.
The same regret bound is concurrently established by Garber and Kretzu [2021] in a similar way.
Moreover, projection-free OCO algorithms, which are able to achieve an O ( √ T ) regret bound for convex losses and an O (log T ) regret bound for strongly convex losses, have been proposed for polyhedral sets [Garber and Hazan, 2016], smooth sets [Levy and Krause, 2019], and special sets that are accessible through a membership oracle [Mhammedi, 2022] or a separation oracle [Garber and Kretzu, 2022], respectively.
Although regret bounds of these algorithms are better than those of OFW, they can only be efﬁciently implemented in some special cases, and thus are still not able to replace OFW.
In addition to these algorithms, Wan and Zhang [2021] show that by setting σ t with a line search technique, OFW can adaptively utilize the strong convexity of decision sets to attain an O ( T 2 / 3 ) regret bound for convex losses and an O ( √ T ) regret bound for strongly convex losses.
Besides the standard setting, projection-free algorithms for other OCO scenarios such as the distributed setting [Zhang et al., 2017, Wan et al., 2020, 2022b] and the bandit setting [Chen et al., 2019, Garber and Kretzu, 2020, 2021] have also been proposed.
However, despite this great ﬂourish of research on projection-free OCO algorithms, the practical problem of delayed feedback has not been considered.
2.2 OCO under delayed feedback
The starting point for studies on OCO under delayed feedback is the seminal work of Weinberger and Ordentlich [2002], which focuses on a special case with a ﬁxed delay d (cid:48) , i.e., the feedback for each decision x t is received at the end of round t + d (cid:48) − 1 .
They propose a technique that can convert any traditional OCO algorithm for the non-delayed setting into this delayed setting.
Speciﬁcally, their technique is to run d (cid:48) instances of a traditional OCO algorithm, where each instance is used every d (cid:48) rounds, which is feasible because the feedback for any decision x t must be available for selecting the decision x t + d (cid:48) .
If the traditional OCO algorithm enjoys a regret bound of R ( T ) in the non-delayed setting, they prove that this technique achieves a regret bound of d (cid:48)
R ( T/d (cid:48) ) .
As a result, by combining with OGD, the regret bound of this technique is O ( √ d (cid:48) T ) for convex losses and O ( d (cid:48) log T ) for strongly convex losses.
However, it needs to run d (cid:48) instances of OGD, which requires more storage and computational resources than the original OGD.
To address this problem, Langford et al.
[2009] study the same delayed setting, and show that simply performing each gradient descent step in the original OGD with a delayed gradient can also achieve the O ( √ d (cid:48) T ) regret bound for convex losses and the O ( d (cid:48) log T ) regret bound for strongly convex losses.
Furthermore, Joulani et al.
[2013] extend the technique in Weinberger and Ordentlich [2002] to handle a more general setting, where each feedback is delayed by arbitrary rounds.
For this setting, their technique is able to attain a regret bound of dR ( T/d ) by combining with a traditional OCO algorithm with R ( T ) regret, where d is the maximum delay.
Note that although this technique can also convert OFW to the delayed setting, it can only attain an O ( d 1 / 4 T 3 / 4 ) regret bound for convex losses and an O ( d 1 / 3 T 2 / 3 ) regret bound for strongly convex losses, which cannot match regret bounds of OFW in the non-delayed setting as long as d is larger than Ω(1) .
Additionally, similar to the technique in Weinberger and Ordentlich [2002], it also needs to run multiple instances of a traditional OCO algorithm, which could be prohibitively resource-intensive.
Many studies [Quanrud and Khashabi, 2015, Joulani et al., 2016, Li et al., 2019, Flaspohler et al., 2021, Wan et al., 2022a] have proposed delayed OCO algorithms, which only require the same storage and computational resources as in the non-delayed setting, but do not consider projection-free algorithms.
Main results
In this section, we ﬁrst introduce necessary preliminaries including the problem setting, deﬁnitions, and assumptions.
Then, we present our delayed OFW and the corresponding theoretical guarantees for convex and strongly convex losses, respectively.
3.1 Preliminaries
We consider the problem of OCO with arbitrary delays [Joulani et al., 2013, Quanrud and Khashabi, 2015].
Similar to the standard OCO, in each round t = 1 , . .
, T , the player ﬁrst chooses a decision x t from the decision set K , and then the adversary selects a convex function f t ( x ) .
However, different from the standard OCO, the gradient g t = ∇ f t ( x t ) is revealed at the end of round t + d t − 1 , where d t ≥ 1 denotes an arbitrary delay.
As a result, the player actually receives gradients { g k | k ∈ F t } at the end of round t , where F t = { k | k + d k − 1 = t } .
Then, we recall the standard deﬁnition for strongly convex functions [Boyd and Vandenberghe, 2004].
Deﬁnition 1 A function f ( x ) : K → R is called β -strongly convex over K if for all x , y ∈ K , it holds that f ( y ) ≥ f ( x ) + (cid:104)∇ f ( x ) , y − x (cid:105) + β 2 (cid:107)
y − x (cid:107) 22 .
Finally, we introduce two assumptions, which are commonly used in studies about OCO [ShalevShwartz, 2011, Hazan, 2016].
Assumption 1 The gradients of all loss functions are bounded by G , i.e., it holds that (cid:107)∇ f t ( x ) (cid:107) 2 ≤
G for any x ∈ K and t ∈ [ T ] .
Assumption 2 The diameter of the decision set K is bounded by D , i.e., it holds that (cid:107) x − y (cid:107) 2 ≤ D for any x , y ∈ K .
3.2 Delayed OFW for convex losses
As presented in (1) and (2), the original OFW for convex losses requires the gradient g t before making the decision x t +1 .
However, in the problem of OCO with arbitrary delays, this requirement is not necessarily satisﬁed, because the player actually receives gradients { g k | k ∈ F t } at each round t , which may not contain g t .
To address this limitation, we propose a natural generalization of OFW to this delayed problem, which is described as follows.
The main idea is to update the decision similar to OFW for each received gradient, and play the latest decision for each round.
In this way, there exist some intermediate decisions that are not really played.
To facilitate presentations, we introduce an additional notation y τ to denote the τ -th intermediate decision.
Moreover, we denote the sum of τ received gradients by ¯ g τ .
Initially, we choose an arbitrary vector y 1 ∈ K and set τ = 1 , ¯ g 0 = 0 .
At each round t = 1 , . . .
, T , we play the latest decision x t = y τ and query the gradient g t = ∇ f t ( x t ) .
Then, we receive delayed gradients that are queried in a set of rounds F t , and perform the following steps for each received gradient.
Speciﬁcally, for any k ∈ F t , inspired by (2) of the original OFW, we ﬁrst compute ¯ g τ = ¯ g τ − 1 + g k and deﬁne
[Equation]
Then, similar to (1) of the original OFW, we perform the following update
[Equation]
where σ τ is a parameter.
Following Wan and Zhang [2021] [Equation] it is set by a line search rule
[Equation]
1: Input: η 2: Initialization: choose an arbitrary vector y 1 ∈ K and set τ = 1 , ¯ g 0 =3: for t = 1 , 2 , . .
, T do 4:
Play x t = y τ and query g t = ∇ f t ( x t ) 5:
Receive a set of delayed gradients { g k | k ∈ F t } 6: for k ∈ F t do 7:
Update ¯ g τ = ¯ g τ − 1 + g k and deﬁne F τ ( y )
=
η (cid:104)
¯ g τ , y (cid:105)
+ (cid:107) y − y 1 (cid:107)8:
Compute v τ ∈ argmin y ∈K (cid:104)∇ F τ ( y τ ) , y (cid:105) 9:
Update y
τ
+
=
y τ
+ σ τ ( v τ − y τ ) with σ τ in (4) and set τ = τ +10: end for 11: end for
Finally, we update τ = τ + 1 so that τ still indexes the latest intermediate decision.
The detailed procedures are summarized in Algorithm 1, which is named as delayed OFW for convex losses.
Let d = max { d 1 , . . .
, d T } .
We establish the following theorem with respect to the regret of
Algorithm 1.
Theorem 1 For any x ∗ ∈ K , under Assumptions 1 and 2, Algorithm 1 with η =
D √ 2 G
( T +2) 3 / 4 has
[Equation]
Theorem 1 shows that without knowing the value of d , our Algorithm 1 can attain an O ( T 3 / 4 + dT 1 / 4 ) regret bound for convex losses with arbitrary delays.
This bound matches the O ( T 3 / 4 ) regret bound of OFW in the non-delayed setting [Hazan, 2016], as long as d does not exceed O ( √ T ) .
Moreover, it is better than the O ( d 1 / 4 T 3 / 4 ) regret bound achieved by combining the technique of Joulani et al.
[2013] and the O ( T 3 / 4 ) regret bound of OFW for convex losses, as long as d does not exceed
O ( T 2 / 3 ) .
3.3 Delayed OFW for strongly convex losses
We proceed to handle β -strongly convex losses by slightly modifying Algorithm 1.
Recall that in the standard OCO without delays, the critical idea of utilizing the strong convexity of losses is to replace the surrogate loss function in (1) by that in (3)
[Wan and Zhang, 2021].
The main difference is that the regularization term in (3) is about all historical decisions, instead of only the initial decision.
Inspired by (3), we ﬁrst redeﬁne F τ ( y ) in Algorithm 1 to
[Equation]
Second, since F τ ( y ) is modiﬁed, we adjust the line search rule to
[Equation]
The detailed procedures are summarized in Algorithm 2, which is named as delayed OFW for strongly convex losses.
Then, we establish the following theorem about the regret of Algorithm 2.
Theorem 2 Suppose all losses are β -strongly convex and Assumptions 1 and 2 hold.
For any x ∗ ∈ K ,
Algorithm 2 has
T (cid:88)
[Equation]
Theorem 2 shows that our Algorithm 2 can attain an O ( T 2 / 3 + d log T ) regret bound for strongly convex losses with arbitrary delays.
First, this bound is better than the O ( T 3 / 4 + dT 1 / 4 ) regret bound
1: Input: β 2: Initialization: choose an arbitrary vector y 1 ∈ K and set τ = 1 , ¯ g 0 =3: for t = 1 , 2 , . .
, T do 4:
Play x t = y τ and query g t = ∇ f t ( x t ) 5:
Receive a set of delayed gradients { g k | k ∈ F t } 6: for k ∈ F t do 7:
Update ¯ g τ = ¯ g τ − 1 + g k and deﬁne F τ ( y )
=
(cid:104)
¯ g τ , y (cid:105)
+ (cid:80) τi
=1 β 2 (cid:107)
y − y i (cid:107)8:
Compute v τ ∈ argmin y ∈K (cid:104)∇ F τ ( y τ ) , y (cid:105) 9:
Update y
τ
+
=
y τ
+ σ τ ( v τ − y τ ) with σ τ in (5) and set τ = τ +10: end for 11: end for in Theorem 1, which is established by only using the convexity condition.
Second, it matches the
O ( T 2 / 3 ) regret bound of OFW for strongly convex losses in the non-delayed setting [Wan and Zhang, 2021], as long as d does not exceed O
( T 2 / 3 / log T ) .
Third, it is better than the O ( d 1 / 3 T 2 / 3 ) regret bound achieved by combining the technique of Joulani et al.
[2013] and the O ( T 2 / 3 ) regret bound of OFW for strongly convex losses, as long as d does not exceed O ( T/ (log T ) 3 / 2 ) .
Theoretical analysis
In this section, we ﬁrst introduce some insights and preliminaries for our analysis, and then prove
Theorems 1 and 2 by introducing some lemmas.
Due to the limitation of space, the proofs for those lemmas are provided in the appendix.
4.1 Insights and preliminaries
Note that in the non-delayed setting, the regret bound of OFW is worse than that of OGD due to the gap between the linear optimization step and the projection operation.
Thus, in the delayed setting, it is natural to conjecture that both the linear optimization step and the delay will affect the regret of our delayed OFW.
To keep the same regret bound as the original OFW, our key insight is to prove that the combined effect of the linear optimization step and the delay can be additive, instead of being multiplicative, and the effect of the delay can be weaker than that of the linear optimization step for a relatively large amount of delay.
Moreover, there are some necessary preliminaries for our analysis.
First, because of the effect of delays, there may exist some gradients that arrive after round T .
Although our Algorithms 1 and 2 do not need to use these gradients, they are useful for the analysis.
As a result, in our analysis, we further set x t = y τ and perform steps 5 to 10 of Algorithms 1 and 2 for any t = T +1 , . .
, T + d − 1 .
In this way, all gradients g 1 , g 2 , . .
. , g T queried by our algorithms are utilized, which produces decisions y 1 , y 2 , . .
, y T +1 .
Then, let τ t = 1 + (cid:80) t − 1 i =1 |F i | for any t ∈
[ T + d ] .
It is not hard to verify that our Algorithms 1 andensure that x t = y τ t (6) for any t ∈
[ T + d − 1] .
Next, we deﬁne
I t = (cid:26) ∅ , if |F t | = 0 , { τ t , τ t + 1 , . . .
, τ t +1 − 1 } , otherwise .
Let s = min { t
| t ∈
[ T + d − 1] , |F t | > 0 } .
It is not hard to verify that ∪
[Equation]
[Equation]
To facilitate the analysis, we further denote the time-stamp of the τ -th gradient used in the update of our Algorithms 1 and 2 by c τ .
To help understanding, one can imagine that our Algorithms 1 andalso implement c τ = k in their step 7 .
If |F t | (cid:54) = 0 , we have { c τ t , . . .
, c τ t +1 − 1 } = F t .
By using this notation, F τ ( y ) deﬁned in Algorithms 1 and 2 are respectively equivalent to
[Equation]
4.2 Proof of Theorem
Let t (cid:48)
=
t + d t − 1 for any t ∈ [ T ] .
According to the convexity of f t ( x ) , we have
[Equation]
where the last equality is due to (6)
[Equation]
Let A = (cid:80)
Tt =1 G
[Equation]
y τ t − y τ t
[Equation]
(cid:107) 2 .
By summing over t = 1 , . . .
, T , we have
[Equation]
Then, we bound the ﬁrst term in the right side of (13) as follows
T
[Equation]
where the ﬁrst equality is due to (9), the second equality is due to i + d i − 1 = t for any i ∈ F t , the third equality is due to (10), and the last equality is due to (7) and (8).
Then, let B =
(cid:80) Tt =1 (cid:104)
g c t , y t − x ∗ (cid:105) and C = (cid:80) T + d − 1 t
=
s (cid:80) τ
t +1 − 1 i = τ t
G (cid:107) y τ t − y
i
(cid:107) 2 .
By combining (13), (14), and (cid:104)
g c i , y τ t − y i (cid:105) ≤ G (cid:107)
y τ t − y
i
(cid:107) 2 , we have
T (cid:88) t = f
[Equation]
( x
[Equation]
(cid:88)
[Equation]
+
[Equation]
+ C.
Next, we proceed to bound terms A , B , and C .
Speciﬁcally, we ﬁrst establish the following bound for the sum of terms A and C by carefully analyzing the distance (cid:107)
y τ t − y τ t (cid:48) (cid:107) 2 in the term A and the distance (cid:107)
y τ t − y
i
(cid:107) 2 in the term C .
Lemma 1 Let y ∗ t = argmin
y ∈K F t − 1 ( y ) for any t ∈
[ T + 1] , where F t ( y ) is deﬁned in (11).
Suppose Assumption 1 and 2 hold, and there exist some constants γ > 0 and 0 < α ≤ 1 such that
F t − 1 ( y t ) − F t − 1 ( y ∗ t ) ≤ γ ( t + 2) − α for any t ∈
[ T + 1] .
Algorithm 1 ensures
[Equation]
where A = (cid:80) T t =1 G (cid:107)
y τ t − y τ t (cid:48) (cid:107) 2 and C =
(cid:80) T + d − 1 t
=
s (cid:80) τ
t +1 − 1 i = τ t
G (cid:107) y τ t − y
i
(cid:107) 2 .
Note that Lemma 1 introduces an assumption about y t and F t − 1 ( y ) .
According to our Algorithm 1, y t is actually generated by approximately minimizing F t − 1 ( y ) with a linear optimization step.
Therefore, by following the analysis of the original OFW [Hazan, 2016], we show that this assumption can be satisﬁed with γ = 8 D 2 and α = 1 / 2 .
Lemma 2 Let y ∗ t = argmin
y ∈K F t − 1 ( y ) for any t ∈
[ T +1] , where F t ( y ) is deﬁned in (11).
Under
Assumptions 1 and 2, for any t ∈
[ T + 1] , Algorithm 1 with η =
[Equation]
Then, by combining η =
[Equation] [Equation]
4 with Lemmas 1 and 2, we have
[Equation]
Furthermore, by following the analysis of the original OFW [Hazan, 2016], we establish an upper bound for the term B .
Lemma 3 Under Assumptions 1 and 2, for any x ∗ ∈ K , Algorithm 1 with η =
[Equation]
[Equation] [Equation]
4 ensures
[Equation]
Finally, by combining (15), (16), and Lemma 3, we complete this proof.
4.3 Proof of Theorem
Since f t ( x ) is β -strongly convex, we have
[Equation]
Then, we note that the ﬁrst term in the right side of (17) can be bounded by reusing (13) and (14).
Speciﬁcally, we have
[Equation]
where terms A and C are deﬁned in the proof of Theorem 1.
Next, we consider the last term in the right side of (18).
For any y t , x t , x ∗ ∈ K , we have
[Equation]
where the last inequality is due to 2 (cid:104)
y t − x t , x t − x ∗
[Equation]
≤
[Equation]
y t − x t (cid:107) 2 (cid:107) x
t − x ∗ (cid:107) 2 and Assumption 2.
Let B (cid:48) =
(cid:80) T t =1 (cid:104) g c t , y t − x ∗ (cid:105)− (cid:80) T t =1 β 2 (cid:107)
y t − x ∗ (cid:107) 22 and E =
(cid:80) T t =1 3 βD 2 (cid:107) y t
− y τ t (cid:107) 2 .
By combining the above inequality and (6) with (18), we have
[Equation]
Then, we proceed to establish upper bounds for terms A , C , and E by carefully analyzing the distance (cid:107) y τ t − y τ t (cid:48) (cid:107) 2 in the term A , the distance (cid:107)
y τ t − y
i
(cid:107) 2 in the term C , and the distance (cid:107) y t − y τ t (cid:107)in the term E .
Lemma 4 Let y ∗ t = argmin
y ∈K F t − 1 ( y ) for any t
= 2 , . .
, T + 1 , where F t ( y ) is deﬁned in (12).
Suppose Assumption 1 and 2 hold, all losses are β -strongly convex, and there exist some constants γ > 0 and 0 ≤ α < 1 such that F t − 1 ( y t ) − F t − 1 ( y ∗ t ) ≤ γ ( t − 1) α for any t
= 2 , . .
, T + 1 .
Algorithm 1 ensures
[Equation]
where A = (cid:80) Tt =
1 G (cid:107)
y τ t − y τ t (cid:48) (cid:107) 2 , C = (cid:80) T + d − 1 t
=
s (cid:80) τ
t +1 − 1 i = τ t
G (cid:107) y τ t − y
i
(cid:107) 2 , and E = (cid:80) Tt =1 3 βD 2 (cid:107) y t − y τ t (cid:107) 2 .
Note that Lemma 4 also introduces an assumption about y t and F t − 1 ( y ) .
According to our Algorithm 2, y t is actually generated by approximately minimizing F t − 1 ( y ) with a linear optimization step.
Therefore, by following the analysis of OFW for strongly convex losses [Wan and Zhang, 2021], we show that this assumption is satisﬁed with γ = 16( G + 2 βD ) 2 /β and α = 1 / 3 .
Lemma 5 Let y ∗ t = argmin
y ∈K F t − 1 ( y ) for any t
= 2 , . .
, T + 1 , where F t ( y ) is deﬁned in (12).
Suppose Assumption 1 and 2 hold, and all losses are β -strongly convex.
For any t = 2 , . . .
, T + 1 ,
Algorithm 2 has
[Equation]
By combining Lemmas 4 and 5, we have
[Equation]
Furthermore, by following the analysis of OFW for strongly convex losses [Wan and Zhang, 2021], we establish an upper bound for the third term in (19).
Lemma 6 Suppose Assumption 1 and 2 hold, and all losses are β -strongly convex.
For any x ∗ ∈ K ,
Algorithm 2 ensures
[Equation]
where B (cid:48)
= (cid:80) Tt =1 (cid:104)
g c t , y t − x ∗ (cid:105) − (cid:80) Tt =1 β 2 (cid:107)
y t − x ∗ (cid:107) 22 .
Finally, by combining (19), (20), and Lemma 6, we complete this proof.
Experiments
In this section, we conduct simulation experiments to verify the performance of our delayed OFW for convex losses and strongly convex losses.
All algorithms are implemented wtih Matlab R2017a and tested on a laptop with 2.4GHz CPU and 16GB memory.
The code is available from https: //github.com/yuanyuwan/NeurIPS22 .
Speciﬁcally, we consider a delayed OCO problem with T = 10000 total rounds over the decision set
K = (cid:8) x ∈ R 1000 (cid:12)(cid:12) (cid:107) x (cid:107) 1 ≤ 200 (cid:9) , which satisﬁes Assumption 2 with D = 400 due to (cid:107) x
− y (cid:107) 2 ≤ (cid:107) x
(cid:107)
+ (cid:107) y (cid:107) 2 ≤ (cid:107) x (cid:107) 1 + (cid:107) y (cid:107) 1 ≤ 400 for any x , y ∈ K .
Inspired by Li et al.
[2019], in each round t ∈ [ T ] , the adversary chooses the following loss function [Equation]
where each element of b t ∈ R 1000 is independently and uniformly sampled from [ − 1 , 1] .
For any x ∈ K , since ∇ f t ( x ) = 2 x + b t , it is easy to verify that (cid:107)∇ f t ( x ) (cid:107) 2 ≤ 2 (cid:107) x (cid:107) 2 + (cid:107) b t (cid:107) 2 ≤ 2 (cid:107) x (cid:107) 1 + (cid:107) b t (cid:107) 2 ≤ 400 + √ 1000 , which implies that this loss function satisﬁes Assumption 1 with G = 400 + √ 1000 .
We also notice that this loss function satisﬁes the deﬁnition of β -strongly convex function (i.e., Deﬁnition 1) with β = 2 .
Moreover, different values of the maximum delay d in the set { 1 , 51 , 101 , . .
. , 501 } are tried in our experiments.
For each speciﬁc d , to simulate arbitrary delays, d t is independently and uniformly sampled from the set {(cid:100) 0 .
1 d (cid:101) , (cid:100) 0 .
1 d (cid:101) + 1 , (cid:100) 0 .
1 d (cid:101) + 2 , . . .
, d } .
We compare our proposed algorithms against the combination of the technique in Joulani et al.
[2013] and OFW.
Our delayed OFW for convex losses and strongly convex losses are denoted as DOFW (i.e.,
Algorithm 1) and DOFW sc (i.e., Algorithm 2), respectively.
Similarly, since the technique in Joulani et al.
[2013] is named as black-box online learning under delayed feedback (BOLD), we denote the combination of BOLD with OFW for convex losses and strongly convex losses as BOLD-OFW and BOLD-OFW sc , respectively.
The parameters of all algorithms are set as what their corresponding theories suggest.
Speciﬁcally, according to our Theorems 1 and 2, we set η =
D √ 2 G
( T +2) 3 / 4 for our DOFW and set β = 2 for our DOFW sc .
According to Joulani et al.
, BOLD-OFW [Figure] and BOLD-OFW sc will maintain several instances of OFW for convex and strongly convex losses, respectively.
Note that in the non-delayed case with d = 1 our delayed OFW actually reduces to the original OFW.
Therefore, our Theorems 1 and 2 can also be utilized to choose the parameters for each instance of OFW maintained in BOLD-OFW and BOLD-OFW sc .
To be precise, in BOLD-OFW, we set η =
[Equation]
3 / 4 for each instance of OFW for convex losses, since the total rounds of each instance is roughly T/d .
In BOLD-OFW sc , we only need to set β = 2 for each instance of OFW for strongly convex losses.
Moreover, for our delayed OFW and each instance of OFW, the initial decision is set to 1 / 50 , where 1 denotes the all-ones vector.
Fig.
1(a) shows the total loss of T rounds for each algorithm under different values of the maximum delay d .
First, when d = 1 , the total loss of our DOFW is the same as that of BOLD-OFW and the total loss of our DOFW sc is the same as that of BOLD-OFW sc , which is reasonable because in this case DOFW and BOLD-OFW reduce to the original OFW for convex losses, and DOFW sc and BOLDOFW sc reduce to the original OFW for strongly convex losses.
Second, for d = 51 , 101 , 151 , . .
, 501 , our DOFW and DOFW sc are better than BOLD-OFW and BOLD-OFW sc respectively, which clearly veriﬁes the advantage of our algorithms in the delayed setting.
It is worthy to notice that d =is larger than T 2 / 3 .
Moreover, for our DOFW and DOFW sc , when d increases from 1 to 501 , the growth of the total loss is very slow, which is consistent with the dependence of our regret bounds on d . Fig. 1(b) shows the cumulative loss for each algorithm when d = 501 .
As the number of iterations increases, the cumulative loss of BOLD-OFW and BOLD-OFW sc increase much faster than that of our algorithms.
Conclusion and future work
In this paper, we propose delayed OFW for OCO with arbitrary delays.
For convex losses, we show that it attains an O ( T 3 / 4 + dT 1 / 4 ) regret bound, which matches the O ( T 3 / 4 ) regret bound of OFW in the non-delayed setting, as long as d does not exceed O ( √ T ) .
When losses are strongly convex, we further prove that it can attain an O ( T 2 / 3 + d log T ) regret bound, which matches the O ( T 2 / 3 ) regret bound of OFW in the non-delayed setting, as long as d does not exceed O ( T 2 / 3 / log T ) .
Simulation experiments demonstrate the performance of delayed OFW in the delayed setting.
This paper only extends the classical OFW to the delayed setting.
In the future, we will investigate how to develop delayed variants for other projection-free online algorithms.
Acknowledgments
This work was partially supported by NSFC (61921006, 62122037), and JiangsuSF (BK20200064).
A Proof of Lemma
We ﬁrst note that F t ( y ) is 2 -strongly convex for any t = 0 , . .
, T , and Hazan and Kale [2012] have proved that for any β -strongly convex function f ( x ) over K and any x ∈ K , it holds that
[Equation]
where x ∗ = argmin x ∈K f ( x ) .
Then, we consider the term
A = (cid:80)
Tt =
1 G (cid:107)
y τ t − y τ t
[Equation]
(cid:107) 2 .
If T ≤ 2 d , we have
[Equation] [Equation]
where the ﬁrst inequality is due to Assumption 2.
If T > 2 d , we have
A =
[Equation]
Because of (21), for any t ∈ [ T + 1] , we have
[Equation]
24) where the last inequality is due to F t − 1 ( y t ) − F t − 1 ( y ∗ t ) ≤ γ ( t + 2) − α .
Moreover, for any i ≥ τ t , we have
[Equation]
where the ﬁrst inequality is still due to (21) and the last inequality is due to Assumption 1.
Because of t (cid:48)
=
t + d t − 1 ≥ t , we have τ t (cid:48) ≥ τ t .
Then, from (25), we have
[Equation]
Then, by substituting (24) and (26) into (23), if T > 2 d , we have
[Equation]
where the second inequality is due to ( τ t + 2) − α/ 2 ≥ ( τ t (cid:48) + 2) − α/ 2 for τ t ≤ τ t (cid:48) and α > 0 .
To bound the second term in the right side of (27), we introduce the following lemma.
Lemma 7 Let τ t = 1 + (cid:80) t − 1 i =1 |F i | for any t ∈
[ T + d ] .
If T > 2 d , for 0 < α ≤ 1 , we have
[Equation]
For the third term in the right side of (27), if T > 2 d , we have
[Equation]
t
[Equation]
t
[Equation]
d
[Equation]
k
[Equation]
t |F k
[Equation]
d
[Equation] [Equation]
k
[Equation]
T
[Equation]
k
[Equation]
t
[Equation] [Equation]
k |F t |
[Equation]
k
[Equation] [Equation]
29) where the second inequality is due to
[Equation]
By substituting (28) and (29) into (27) and combining with (22), we have
[Equation]
[Equation]
Then, for the term C =
[Equation]
y
[Equation]
y
[Equation]
2 , we have
C =
[Equation]
+
[Equation]
k
[Equation]
where the ﬁrst inequality is due to Assumption 2, the second inequality is due to (24) and (25), and the third inequality is due to ( τ t + 2) − α/ 2 ≥ ( i + 2) − α/ 2 for τ t ≤ i and α > 0 .
Moreover, for any t ∈ [ T + d − 1] and k ∈ F t , since 1 ≤ d k ≤ d , we have
[Equation]
which implies that
[Equation]
Then, it is easy to verify that
[Equation]
Therefore, by combining with (31)
[Equation]
we have
[Equation]
Furthermore
[Equation]
we introduce the following lemma.
Lemma 8 Let τ t = 1 + (cid:80) t − 1 i =1 |F i | for any t ∈
[ T + d ] and s = min { t | t ∈
[ T + d − 1] , |F t | > 0 } .
For 0 < α ≤ 1 , we have
[Equation]
By substituting (34) into (33), we have
[Equation]
We complete the proof by combing (30) and (35).
B Proof of Lemma
At the beginning of this proof, we recall the standard deﬁnition for smooth functions [Boyd and
Vandenberghe, 2004].
Deﬁnition
A function f ( x ) : K → R is called α -smooth over K if for all x , y ∈ K , it holds that f ( y ) ≤ f ( x ) + (cid:104)∇ f ( x ) , y − x (cid:105) + α 2 (cid:107) y − x (cid:107) 22 .
It is not hard to verify that F t ( y ) is 2 -smooth over K for any t ∈
[ T ] .
This property will be utilized in the following.
For brevity, we deﬁne h t =
F t − 1 ( y t ) − F t − 1 ( y ∗ t ) for t = 1 , . .
, T +1 and h t ( y t − 1 ) =
F t − 1 ( y t − 1 ) −
F t − 1 ( y ∗ t ) for t = 2 , . .
, T + 1 .
For t = 1 , since y 1 = argmin y ∈K (cid:107) y − y 1 (cid:107) 22 , we have
[Equation]
Then, for any T + 1 ≥ t ≥ 2 , we have
[Equation]
where the ﬁrst inequality is due to y ∗ t − 1 = argmin y ∈K F t − 2 ( y ) and the last inequality is due to
Assumption
[Equation]
Moreover, for any T + 1 ≥ t ≥ 2 , we note that F t − 2 ( x ) is also 2 -strongly convex, which implies that
[Equation]
[Equation]
where the ﬁrst inequality is due to (21).
Similarly, for any T + 1 ≥ t ≥
[Equation]
which implies that
[Equation]
By combining (37), (38), and (39), for any T + 1 ≥ t ≥ 2 , we have
[Equation]
Then, for any T + 1 ≥ t ≥ 2 , since F t − 1 ( y ) is 2 -smooth, we have
[Equation]
Moreover, for any t ∈ [ T ] , according to Algorithm 1, we have
[Equation]
Therefore, for t = 2 , by combining (40) and (41), we have
[Equation]
where the second inequality is due to (42), and the ﬁrst equality is due to (36) and η =
D √ 2 G
( T +2)
[Equation]
Then, for any t = 3 , . .
, T + 1 , by deﬁning σ (cid:48)
t − 1 = 2 / √ t + 1 and assuming h t − 1 ≤ 8 D 2 √ t +1 , we have
[Equation]
[Equation]
where the ﬁrst inequality is due to (41) and (42), the second inequality is due to v t − 1 ∈ argmin y ∈K (cid:104)∇ F t − 1 ( y t − 1 ) , y (cid:105) , the third inequality is due to the convexity of F t − 1 ( y ) , the fourth inequality is due to (40), and the last inequality is due to
[Equation]
for any t ≥ 0 .
Note that (45) can be derived by dividing ( t + 1) √ t + 2 into both sides of the following inequality √
[Equation]
By combining (36), (43), and (44), we complete this proof.
C Proof of Lemma
In the beginning, we deﬁne y ∗ t = argmin
y ∈K F t − 1 ( y ) for any t ∈
[ T + 1] , where F t ( y ) =
[Equation] [Equation]
Then, it is easy to verify that
[Equation]
Therefore, we will continue to upper bound the right side of (46).
By applying Lemma 2, we have
[Equation]
where the second inequality is due to (21) and Assumption 1, and the last inequality is due to (cid:80)
[Equation]
Then, to bound (cid:80) Tt =1 (cid:104) g c t , y ∗ t − x ∗ (cid:105) , we introduce the following lemma.
Lemma 9 (Lemma 6.6 of Garber and Hazan [2016]) Let { f t ( y ) }
Tt =1 be a sequence of loss functions and let y ∗ t ∈ argmin y ∈K (cid:80) ti =
f
i
( y ) for any t ∈
[ T ] .
Then, it holds that
[Equation]
To apply Lemma 9, we deﬁne ˜ f 1 ( y )
=
η (cid:104) g c 1 , y (cid:105)
+ (cid:107) y − y 1 (cid:107) 22 and
˜ f t ( y ) =
η (cid:104)
g c t , y (cid:105) for any t ≥ 2 .
Note that F t ( y ) =
(cid:80) ti =1 ˜
f i ( y ) and y ∗ t +1 = argmin
y ∈K F t ( y ) for any t
= 1 , . .
, T .
Then, by applying Lemma 9 to { ˜ f t ( y ) } Tt =1 , we have
[Equation]
which implies that
[Equation]
According to Assumption 2, we have
[Equation]
Then, we have
T
[Equation] [Equation]
where the second inequality is due to (39) and Assumption 1, and the last inequality is due to η =
D √
[Equation]
By substituting (47) and (48) into (46), we complete the proof.
D Proof of Lemma
We ﬁrst consider the term E =
(cid:80) Tt =
[Equation]
βD 2 (cid:107) y t − y τ t (cid:107) 2 .
If T ≤ 2 d , it is easy to verify that
[Equation]
where the ﬁrst inequality is due to Assumption 2.
Then, if T > 2 d , we have
[Equation]
Because F t − 1 ( y ) is ( t − 1) β -strongly convex for any t = 2 , . .
, T + 1 , we have
[Equation]
) where the ﬁrst inequality is due to (21) and the second inequality is due to F t − 1 ( y t ) − F t − 1 ( y ∗ t ) ≤ γ ( t − 1) α .
Before considering (cid:107) y ∗ t − y ∗ τ t (cid:107) 2 , we deﬁne ˜
f t ( y ) =
(cid:104) g c t , y (cid:105)
+ β 2 (cid:107) y − y t (cid:107) 22 for any t = 1 , . . .
, T .
Note that F t ( y ) =
(cid:80) ti =1 ˜
f i ( y ) .
Moreover, for any x , y ∈ K and t = 1 , . .
, T , we have
[Equation]
where the last inequality is due to Assumptions 1 and 2.
Because of (21)
[Equation]
for any i ≥ j >
[Equation]
we have
[Equation]
where the last inequality is due to y ∗ j = argmin y ∈K F j − 1 ( y ) and (52).
Note that all gradients queried at rounds 1 , . . .
, t − d must arrive before round t .
Therefore, for any t ≥ 2 d + 1 , we have τ t = 1 + (cid:80) t − 1 k =1 |F k | ≥ t − d + 1 > t − d and
[Equation]
where the ﬁrst inequality is due to t ≥ τ t > 1 and (53)
[Equation]
By combining (50) with (51) and (54), if T > 2 d , we have
[Equation]
where the second inequality is due to ( τ t − 1) 1 − α ≤ ( t − 1) 1 − α for t ≥ τ t > 1 and α < 1 , and the last inequality is due to Lemma 7 and 0 < 1 − α ≤ 1 .
By combining (49) with the above inequality, we have
[Equation]
Then, we proceed to bound the term C = (cid:80) T + d − 1 t = s
[Equation]
i
[Equation]
G (cid:107) y τ t − y
i
(cid:107) 2 .
Similar to (31), we ﬁrst have
[Equation]
By combining (55) with |F s | ≤ d , (51), and (53), we have
[Equation]
+
[Equation]
where the ﬁrst inequality is due to ( τ t − 1) 1 − α ≤ ( i − 1) 1 − α for 0 < τ t − 1 ≤ i − 1 and α < 1 , and the last inequality is due to Lemma 8, 0 < 1 − α ≤ 1 , and i − τ t ≤ τ t +1 − 1 − τ t ≤ |F t | ≤ d .
Recall that we have deﬁned
I
[Equation]
∅
[Equation]
if |F t | =
[Equation]
{ τ t , τ t + 1 , . . .
, τ t +1 − 1 } , otherwise .
It is not hard to verify that
[Equation]
By combining (57) with (56), we have
[Equation]
Next, we proceed to bound the term
A = (cid:80) Tt
=1 G (cid:107)
y τ t − y τ t (cid:48) (cid:107) 2 .
Similar to (23), if T > 2 d , we have
A
[Equation]
(cid:88)
[Equation] [Equation]
where the second inequality is due to (51) and (53), and the last inequality is due to τ t (cid:48) ≥ τ t > 1 and ( τ t (cid:48)
− τ t
[Equation] [Equation]
(cid:80)
[Equation]
Then, we introduce the following lemma.
Lemma 10 Let h k
=
(cid:80) ki
=1 |F i | .
If T > 2 d , we have
[Equation]
By applying Lemmas 7 and 10 to (59) and combining with (22), we have
[Equation]
Finally, by combining (58) and (60), we complete this proof.
E Proof of Lemmas 5 and
Recall that F τ ( y ) deﬁned in Algorithm 2 is equivalent to that deﬁned in (12).
Let ˜ f t ( y )
= (cid:104) g c t , y (cid:105)
+ β 2 (cid:107) y − y t (cid:107) 22 for any t = 1 , . . .
, T , which is β -strongly convex.
Moreover, as proved in (52), functions ˜ f 1 ( y ) , . .
˜ f T ( y ) are ( G + βD ) -Lipschitz over K (see the deﬁnition of Lipschitz functions in Hazan [2016]).
Then, because of ∇ ˜ f t ( y t ) =
g c t , it is not hard to verify that decisions y 1 , . . .
, y T +1 in our
Algorithm 2 are actually generated by performing OFW for strongly convex losses (see Algorithmin Wan and Zhang [2021] for details) on functions ˜ f 1 ( y ) , . . . ,
˜ f T ( y ) .
Note that when Assumptionholds, and functions ˜ f
( y ) , . . . ,
˜ f T ( y ) are β -strongly convex and G (cid:48)
-Lipschitz, Lemma 6 of Wan and Zhang [2021] has already shown that
[Equation]
for any t = 2 , . . .
, T +1 .
Therefore, our Lemma 5 can be derived by simply substituting G (cid:48)
=
G + βD into the above inequality.
Moreover, when Assumption 2 holds, and functions ˜ f 1 ( y ) , . .
˜ f T ( y ) are β -strongly convex and
G (cid:48) -Lipschitz, Theorem 3 of Wan and Zhang [2021] has already shown that
[Equation]
We notice that (cid:80)
[Equation]
=
[Equation]
=
[Equation]
Tt =1 ˜ f t ( x ∗ ) .
Therefore, our Lemma 6 can be derived by simply substituting G (cid:48)
=
G + βD into the above inequality.
F Proof of Lemma
Since the gradient g 1 must arrive before round d + 1 , for any T ≥ t ≥ 2 d + 1 , it is easy to verify that τ t
= 1 + (cid:80) t − 1 i =1 |F i | ≥ 1 + (cid:80) d
+1 i =1 |F
i | ≥ 2 .
Moreover, for any i ≥ 2 and ( i + 1) d ≥ t ≥ id + 1 , since all gradients queried at rounds 1 , . . .
, ( i − 1) d + 1 must arrive before round id + 1 , we have τ t = 1 + t
[Equation]
1 (cid:88) i = |F
[Equation]
|
[Equation]
( i
[Equation]
1) d + 2 .
[Equation]
Then
[Equation]
we have
[Equation]
where the ﬁrst inequality is due to ( τ t − 1) − α/ 2 ≤ 1 for α > 0 and τ t ≥ 2 , and the second inequality is due to (61) and α > 0 .
G Proof of Lemma
Because of τ t = 1 + (cid:80) t − 1 i =1 |F i | , we have
T
[Equation]
1 (cid:88)
[Equation] [Equation]
[Equation] [Equation]
d
[Equation] [Equation]
(62) where the ﬁrst inequality is due to (32) and ( (cid:80) t − 1 i = s |F
i | )
[Equation] [Equation]
≤ ( (cid:80) ti
[Equation]
s |F
i | ) α/ 2 .
Let h t = (cid:80) ti
= s |F
i | for any t = s, . .
, T + d − 1 .
Since 0 < α ≤ 1 , it is not hard to verify that
[Equation]
Finally, we complete this proof by combining (62) with (63).
H Proof of Lemma
It is not hard to verify that
[Equation]
where the ﬁrst inequality is due to s ≤ d < 2 d + 1 , and the second inequality is due to t (cid:48) − 1 = t + d t − 2 < t + d − 1 .
Moreover, we have
[Equation]
+
[Equation]
where the last equality is due to h s =
|F s | and h T + d − 1 = T .
Finally, we complete this proof by combining the above two inequalities.