Online Frank-Wolfe with Arbitrary Delays
Abstract
The online Frank-Wolfe (OFW) method has gained much popularity for onlineconvex optimization due to its projection-free property.
Previous studies showedthat OFW can attain an O ( T 3 / 4 ) regret bound for convex losses and an O ( T 2 / 3 )regret bound for strongly convex losses.
However, they assumed that each gradientqueried by OFW is revealed immediately, which may not hold in practice and limitsthe application of OFW.
To address this limitation, we proposed a delayed variantof OFW, which allows gradients to be delayed by arbitrary rounds.
The main idea isto perform an update similar to OFW after receiving any delayed gradient, and playthe latest decision for each round.
Despite its simplicity, we prove that our delayedvariant of OFW is able to achieve an O ( T 3 / 4 + dT 1 / 4 ) regret bound for convexlosses and an O ( T 2 / 3 + d log T ) regret bound for strongly convex losses, where dis the maximum delay.
This is somewhat surprising since under a relatively largeamount of delay (e.g., d = O ( √ T ) for convex losses and d = O ( T 2 / 3 / log T ) forstrongly convex losses), the delayed variant of OFW enjoys the same regret boundlike that of the original OFW.
Introduction
Online convex optimization (OCO) has become a leading paradigm for online learning due to its capability to model various problems from diverse domains such as online routing, online collaborativeﬁltering and online advertisement [Hazan, 2016].
In general, it is formulated as a structured repeatedgame between a player and an adversary.
In each round t , the player ﬁrst chooses a decision x t froma convex decision set K ⊆ R n , where n is the dimensionality.
Then, the adversary selects a convexfunction f t ( x ) : K (cid:55)→ R and the player suffers a loss f t ( x t ) .
The player aims to choose decisionssuch that the regret
R ( T ) =
T (cid:88) t =
f t ( x t ) − min x ∈K
T (cid:88) t =
f t ( x ) is sublinear in the number of total rounds T .
Online gradient descent (OGD) is a standard method for
OCO, which enjoys an O ( √ T ) regret bound for convex losses [Zinkevich, 2003] and an O (log T )regret bound for strongly convex losses [Hazan et al., 2007].
However, it needs to compute aprojection onto the decision set to ensure the feasibility of each decision, which is computationallyexpensive for complex decision sets [Hazan and Kale, 2012].
To tackle this computational issue, Hazan and Kale [2012] proposed the online Frank-Wolfe (OFW)method, which has become one of the most commonly used algorithms for OCO over complexdecision sets.
The main advantage of OFW is its projection-free property: instead of performing theprojection operation, it utilizes a linear optimization step to select a feasible decision, which couldbe much more efﬁcient.
For example, in the problem of online collaborative ﬁltering, the decisionset consists of matrices with a bounded trace norm, and the linear optimization step is at least an order of magnitude faster than the projection operation [Hazan and Kale, 2012].
Moreover, it hasbeen showed that OFW achieves an O ( T 3 / 4 ) regret bound for convex losses [Hazan and Kale, 2012,
Hazan, 2016] and an O ( T 2 / 3 ) regret bound for strongly convex losses [Wan and Zhang, 2021, Garberand Kretzu, 2021], which are the best known regret bounds of projection-free methods without furtherassumptions.
However, OFW requires that the gradient ∇ f t ( x t ) is revealed immediately after making the decisionx t , which is not necessarily satisﬁed in reality.
For example, in the previously mentioned onlinecollaborative ﬁltering [Hazan and Kale, 2012], the decision is a prediction of a user-item rating matrix,and the corresponding gradient depends on the true rating of a user on a item, which may not bedecided by the user immediately.
Therefore, it is natural to consider a more practical setting, wherethe gradient ∇ f t ( x t ) arrives at the end of round t + d t − 1 , and d t ≥ 1 denotes an arbitrary delay.
To handle this setting, one potential way is to combine OFW with an existing black-box techniquefor converting any traditional OCO algorithm into this delayed setting [Joulani et al., 2013].
To beprecise, this black-box technique is to pool independent instances of OFW, each of which acts as alearner in the non-delayed setting over a subsequence of rounds.
In each round, a single instance willbe taken out from the pool, which makes a decision and then waits for its feedback before rejoiningthe pool.
If the pool is empty, a new instance of OFW will be added to it.
Moreover, accordingto Joulani et al.
[2013], their black-box technique is able to attain a regret bound of dR ( T/d ) bycombining with a traditional OCO algorithm with R ( T ) regret, where d is the maximum delay.
Asa result, combining it with OFW will attain an O ( d 1 / 4 T 3 / 4 ) regret bound for convex losses and an
O ( d 1 / 3 T 2 / 3 ) regret bound for strongly convex losses, which magnify the regret bounds of OFW inthe non-delayed setting by a coefﬁcient about the delay.
Thus, it is natural to ask whether the effectabout delay can be further reduced.
In this paper, we give an afﬁrmative answer by developing a simple method called delayed OFW,which is able to be robust to a relatively large amount of delay for both convex and strongly convexlosses.
Different from the black-box technique that needs to maintain multiple instances of OFW[Joulani et al., 2013], our delayed OFW is a natural extension of OFW in the delayed setting, whichupdates the decision similar to OFW after receiving any delayed gradient, and plays the latest decisionfor each round.
Our theoretical contributions are summarized as follows.
• First, we prove that our delayed OFW attains an O ( T 3 / 4 + dT 1 / 4 ) regret bound for convexlosses, where d is the maximum delay, which matches the O ( T 3 / 4 ) regret bound in thenon-delayed setting as long as d does not exceed O ( √ T ) .
• Second, we prove that our delayed OFW attains an O ( T 2 / 3 + d log T ) regret bound forstrongly convex losses, which matches the O ( T 2 / 3 ) regret bound in the non-delayed settingas long as d does not exceed O ( T 2 / 3 / log T ) .
Therefore, our regret bounds are strictly better than those achieved by combining the black-boxtechnique [Joulani et al., 2013] with OFW, when the term about d in them is not dominant.
Related work
In this section, we brieﬂy review related work on projection-free algorithms for OCO, and OCOunder delayed feedback.
2.1 Projection-free algorithms for OCO
The OFW method [Hazan and Kale, 2012, Hazan, 2016] is the ﬁrst projection-free algorithm for
OCO, which is an online extension of the classical Frank-Wolfe method [Frank and Wolfe, 1956,
Jaggi, 2013].
For convex losses, OFW ﬁrst chooses an arbitrary x 1 ∈ K , and then iteratively updatesits decision by the following linear optimization step
[Equation]
where F t ( x ) is a surrogate loss function deﬁned as
[Equation]
and η, σ t are two parameters.
With appropriate parameters, it can attain an O ( T 3 / 4 ) regret bound forconvex losses.
If losses are convex and smooth, Hazan and Minasyan [2020] proposed a randomized projection-freemethod, which is based on a classical OCO method called follow the perturbed leader [Kalai and
Vempala, 2005], and achieved a regret bound of O
( T 2 / 3 ) .
Recently, Wan and Zhang [2021] provedthat OFW can achieve an O ( T 2 / 3 ) regret bound for strongly convex losses.
Speciﬁcally, to utilize thestrong convexity of losses, Wan and Zhang [2021] redeﬁned F t ( x ) in (2) to
[Equation]
where β is the modulus of the strong convexity.
The same regret bound was concurrently establishedby Garber and Kretzu [2021].
Moreover, projection-free OCO algorithms, which are able to achieve an O ( √ T ) regret bound forconvex losses and an O (log T ) regret bound for strongly convex losses, were proposed for polyhedralsets [Garber and Hazan, 2016], smooth sets [Levy and Krause, 2019], and special sets that areaccessible through a membership oracle [Mhammedi, 2021] or a separation oracle [Garber and
Kretzu, 2022], respectively.
Although the regret bounds of these algorithms are better than those of
OFW, they can only be efﬁciently implemented in some special sets, and thus are still not able toreplace OFW.
Different from these algorithms limited to special sets, Wan and Zhang [2021] showedthat by setting σ t with a line search technique, OFW can adaptively utilize the strong convexity ofdecision sets to attain an O ( T 2 / 3 ) regret bound for convex losses and an O ( √ T ) regret bound forstrongly convex losses.
Despite great progresses about projection-free OCO algorithms, these studies did not consider thepractical problem of delayed feedback.
2.2 OCO under delayed feedback
The ﬁrst work about OCO under delayed feedback was due to Weinberger and Ordentlich [2002],which focused on a special case with a ﬁxed delay d (cid:48) , i.e., the feedback for each decision x t isreceived at the end of round t + d (cid:48) − 1 .
They proposed a technique that can convert any traditional
OCO algorithm for the non-delayed setting into this delayed setting.
Speciﬁcally, their techniqueis to run d (cid:48) instances of a traditional OCO algorithm, where each instance is used every d (cid:48) rounds,which is feasible because the feedback for any decision x t must be available for selecting the decisionx t + d (cid:48) .
If the traditional OCO algorithm enjoys a regret bound of R ( T ) in the non-delayed setting,they proved that this technique achieves a regret bound of d (cid:48)
R ( T/d (cid:48) ) .
As a result, by combiningwith OGD, the regret bound of this technique is O ( √ d (cid:48) T ) for convex losses and O ( d (cid:48) log T ) forstrongly convex losses.
However, it needs to run d (cid:48) instances of OGD, which requires more storageand computational resources than the original OGD.
To address this problem, Langford et al.
[2009]studied the same delayed setting, and showed that simply performing each gradient descent step inthe original OGD with a delayed gradient can also achieve the O ( √ d (cid:48) T ) regret bound for convexlosses and the O ( d (cid:48) log T ) regret bound for strongly convex losses.
Furthermore, Joulani et al.
[2013] extended the technique in Weinberger and Ordentlich [2002] tohandle a more general setting, where each feedback is delayed by arbitrary rounds.
For this setting,their technique is able to attain a regret bound of dR ( T/d ) by combining with a traditional OCOalgorithm with R ( T ) regret, where d is the maximum delay.
Note that although this technique canalso convert OFW to the delayed setting, it can only attain an O ( d 1 / 4 T 3 / 4 ) regret bound for convexlosses and an O ( d 1 / 3 T 2 / 3 ) regret bound for strongly convex losses, which cannot match regretbounds of OFW in the non-delayed setting as long as d is larger than O (1) .
Additionally, similarto the technique in Weinberger and Ordentlich [2002], it also needs to run multiple instances of atraditional OCO algorithm, which could be prohibitively resource-intensive.
Many studies [Quanrud and Khashabi, 2015, Joulani et al., 2016, Li et al., 2019, Wan et al., 2022, Flaspohler et al., 2021]proposed delayed OCO algorithms, which only require the same storage and computational resourcesas in the non-delayed setting, but did not consider the projection-free algorithms.
Main results
In this section, we ﬁrst introduce necessary preliminaries including the problem setting, deﬁnitions,and assumptions.
Then, we present our delayed OFW and the corresponding theoretical guaranteesfor convex and strongly convex losses, respectively.
3.1 Preliminaries
We consider the problem of OCO with arbitrary delays [Joulani et al., 2013, Quanrud and Khashabi,2015].
Similar to the standard OCO, in each round t = 1 , . .
, T , the player ﬁrst chooses a decisionx t from the decision set K , and then the adversary selects a convex function f t ( x ) .
However, differentfrom the standard OCO, the gradient g t = ∇ f t ( x t ) is revealed at the end of round t + d t − 1 , whered t ≥ 1 denotes an arbitrary delay.
As a result, the player actually receives gradients { g k | k ∈ F t } atthe end of round t , where
[Equation]
Then, we recall the standard deﬁnition for strongly convex functions [Boyd and Vandenberghe, 2004].
Deﬁnition 1 A function f ( x ) : K → R is called β -strongly convex over K if it holds that
[Equation]
for all x , y ∈ K .
Finally, we introduce two assumptions, which are commonly used in studies about OCO [ShalevShwartz, 2011, Hazan, 2016].
Assumption 1 The gradients of all loss functions are bounded by G , i.e., it holds that (cid:107)∇ f t ( x ) (cid:107) 2 ≤
G for any x ∈ K and t ∈ [ T ] .
Assumption 2 The diameter of the decision set K is bounded by D , i.e., it holds that (cid:107) x − y (cid:107) 2 ≤ Dfor any x , y ∈ K .
3.2 Delayed OFW for convex losses
As presented in (1) and (2), the original OFW for convex losses requires the gradient g t beforemaking the decision x t +1 .
However, in the problem of OCO with arbitrary delays, this requirementis not necessarily satisﬁed, because the player actually receives gradients { g k | k ∈ F t } at each roundt , which may not contain g t .
To address this limitation, we propose a natural generalization of OFWto this delayed problem, which is described as follows.
The main idea is to update the decision similar to OFW for each received gradient, and play thelatest decision for each round.
In this way, there exist some intermediate decisions that are notreally played.
To facilitate presentations, we introduce an additional notation y τ to denote the τ -thintermediate decision.
Moreover, we denote the sum of τ received gradients by ¯ g τ .
Initially, wechoose an arbitrary vector y 1 ∈ K and set τ = 1 , ¯ g 0 = 0 .
At each round t = 1 , . . .
, T , we play thelatest decision x t = y τ and query the gradient g t = ∇ f t ( x t ) .
Then, we receive delayed gradientsthat are queried in a set of rounds F t , and perform the following steps for each received gradient.
Speciﬁcally, for any k ∈ F t , inspired by (2) of the original OFW, we ﬁrst compute ¯ g τ = ¯ g τ − 1 + g kand deﬁne
[Equation]
Then, similar to (1) of the original OFW, we perform the following update
[Equation]
1: Input: η 2: Initialization: choose an arbitrary vector y 1 ∈ K and set τ = 1 , ¯ g 0 =3: for t = 1 , 2 , . .
, T do 4:
Play x t = y τ and query g t = ∇ f t ( x t ) 5:
Receive a set of delayed gradients { g k | k ∈ F t } 6: for k ∈ F t do 7: ¯ g τ = ¯ g τ − 1 + g k 8:
Deﬁne F τ ( y ) =
η (cid:104)
[Equation]
τ , y (cid:105) + (cid:107)
[Equation]
(cid:107)9: v τ ∈ argmin y ∈K (cid:104)∇ F τ ( y τ ) , y (cid:105) 10: y τ +
=
y τ
+ σ τ ( v τ − y τ ) with σ τ in (4) 11: τ = τ
+12: end for 13: end for where σ τ is a parameter.
Following Wan and Zhang [2021], it is set by a line search rule
[Equation]
(cid:107)
[Equation]
Finally, we update τ = τ + 1 so that τ still indexes the latest intermediate decision.
The detailed procedures are summarized in Algorithm 1, which is named as delayed OFW for convexlosses.
Deﬁne
[Equation]
We ﬁrst establish the following theorem with respect to the regret of Algorithm 1.
Theorem 1 For any x ∗ ∈ K , under Assumptions 1 and 2, Algorithm 1 with η =
D √ 2 G
( T +2) 3 / 4 has
T (cid:88) t =
[Equation] [Equation]
[Equation]
Theorem 1 shows that our Algorithm 1 can achieve an O ( T 3 / 4 + dT 1 / 4 ) regret bound for convexlosses with arbitrary delays.
This bound matches the O ( T 3 / 4 ) regret bound of OFW in the nondelayed setting [Hazan and Kale, 2012, Hazan, 2016], as long as d does not exceed O ( √ T ) .
Moreover,it is better than the O ( d 1 / 4 T 3 / 4 ) regret bound achieved by combining the technique of Joulani et al.[2013] and the O ( T 3 / 4 ) regret bound of OFW for convex losses, as long as d does not exceed
O ( T 2 / 3 ) .
3.3 Delayed OFW for strongly convex losses
We proceed to handle β -strongly convex losses by slightly modifying Algorithm 1.
Recall that in thestandard OCO without delays, the critical idea of utilizing the strong convexity of losses is to replacethe surrogate loss function in (1) by that in (3)
[Wan and Zhang, 2021].
The main difference is thatthe regularization term in (3) is about all historical decisions, instead of only the initial decision.
Inspired by (3), we ﬁrst redeﬁne F τ ( y ) in Algorithm 1 to
[Equation]
Second, since F τ ( y ) is modiﬁed, we adjust the line search rule to
[Equation]
The detailed procedures are summarized in Algorithm 2, which is named as delayed OFW for stronglyconvex losses.
Then, we establish the following theorem about the regret of Algorithm 2.
1: Input: β 2: Initialization: choose an arbitrary vector y 1 ∈ K and set τ = 1 , ¯ g 0 =3: for t = 1 , 2 , . .
, T do 4:
Play x t = y τ and query g t = ∇ f t ( x t ) 5:
Receive a set of delayed gradients { g k | k ∈ F t } 6: for k ∈ F t do 7: ¯ g τ = ¯ g τ − 1 + g k 8:
Deﬁne F τ ( y ) =
(cid:104)
¯ g τ , y (cid:105)
+ (cid:80) τi
=1 β 2 (cid:107)
y − y i (cid:107)9: v τ ∈ argmin y ∈K (cid:104)∇ F τ ( y τ ) , y (cid:105) 10:
y τ +1 =
y τ
+ σ τ ( v τ − y τ ) with σ τ in (5) 11: τ = τ
+12: end for 13: end for
Theorem 2 For any x ∗ ∈ K , suppose Assumptions 1 and 2 hold, and all losses are β -strongly convex,
Algorithm 2 has
[Equation]
Theorem 2 shows that our Algorithm 2 can achieve an O ( T 2 / 3 + d log T ) regret bound for stronglyconvex losses with arbitrary delays.
First, this bound is better than the O ( T 3 / 4 + dT 1 / 4 ) regret boundin Theorem 1, which is established by only using the convexity condition.
Second, it matches the
O ( T 2 / 3 ) regret bound of OFW for strongly convex losses in the non-delayed setting [Garber and
Kretzu, 2021, Wan and Zhang, 2021], as long as d does not exceed O
( T 2 / 3 / log T ) .
Third, it is betterthan the O ( d 1 / 3 T 2 / 3 ) regret bound achieved by combining the technique of Joulani et al.
[2013] andthe O ( T 2 / 3 ) regret bound of OFW for strongly convex losses, as long as d does not exceed O ( T 2 / 3 ) .
Theoretical analysis
In this section, we ﬁrst introduce necessary preliminaries for our analysis, and then prove Theorem1 by introducing some lemmas.
Due to the limitation of space, the proofs for those lemmas and Theorem 2 are provided in the supplementary material.
4.1 Preliminaries
Because of the effect of delays, there may exist some gradients that arrive after round T .
Althoughour Algorithms 1 and 2 do not need to use these gradients, they are useful for the analysis.
As aresult, in our analysis, we further set x t = y τ and perform steps 5 to 12 of Algorithms 1 and 2 forany t
= T + 1 , . .
, T + d − 1 .
In this way, all gradients g 1 , g 2 , . .
. , g T queried by our algorithmsare utilized, which produces decisions y 1 , y 2 , . .
, y T +1 .
Then, let τ t = 1 + (cid:80) t − 1 i =1 |F i | for any t ∈
[ T + d ] .
It is not hard to verify that our Algorithms 1 and 2ensure thatx t =
y
[Equation]
t (6) for any t ∈
[ T + d − 1] .
Next, we deﬁne
I t = (cid:26) ∅ , if |F t | = 0 , { τ t , τ t + 1 , . . .
, τ t +1 − 1 } , otherwise .
Let s = min { t
| t ∈
[ T + d − 1] , |F t | > 0 } .
It is not hard to verify that∪ T + d − 1 t
=
s
I t =
[ T ] and I i ∩ I j = ∅ , ∀ i (cid:54) =
j (8) and∪ T + d − 1 t = s
F t =
[ T ] and F i ∩ F j = ∅ , ∀ i (cid:54) =
j.
To facilitate the analysis, we further denote the time-stamp of the τ -th gradient used in the update ofour Algorithms 1 and 2 by c τ .
To help understanding, one can imagine that our Algorithms 1 and 2also implement c τ = k in their step 7 .
If |F t | (cid:54) = 0 , we have
[Equation]
Moreover, by using this notation, F τ ( y ) deﬁned in Algorithms 1 and 2 is respectively equivalent to
[Equation]
4.2 Proof of Theorem 1
Let t (cid:48)
=
t + d t − 1 for any t ∈ [ T ] .
According to the convexity of f t ( x ) , we have
[Equation]
where the last equality is due to (6).
By summing over t = 1 , . . .
, T , we have
T (cid:88) t =
[ f t ( x t ) − f t ( x ∗ )] ≤
T (cid:88) t
= (cid:104) g
[Equation]
, x
[Equation]
−
[Equation]
(cid:105) ≤
T (cid:88) t
= (cid:104) g t , y τ t (cid:48) − x ∗ (cid:105) +
T (cid:88) t =
G (cid:107) y τ t − y τ t (cid:48) (cid:107)(cid:124) (cid:123)(cid:122) (cid:125) :
= A .
Then, we bound the ﬁrst term in the right side of (13) as follows
T (cid:88) t
= (cid:104) g
[Equation]
y
[Equation]
−
[Equation]
(cid:105) =
[Equation]
(cid:88) t
[Equation]
s (cid:88) i ∈F t (cid:104) g
[Equation]
y
[Equation]
i
[Equation]
di − 1 −
[Equation]
(cid:105) =
[Equation]
(cid:88) t
[Equation]
s (cid:88) i ∈F t (cid:104)
[Equation]
(cid:105) =
T
[Equation]
(cid:88) t =
s τ t
+1 − 1 (cid:88) i =
τ t (cid:104)
[Equation]
c i
[Equation]
y
[Equation]
t − x
[Equation]
(cid:105) =
T
[Equation]
d
[Equation]
(cid:88) t =
s τ t +
[Equation]
(cid:88)
i
[Equation]
t ( (cid:104) g
[Equation]
i
[Equation]
y i − x
[Equation]
(cid:105)
[Equation]
(cid:104)
g c i
[Equation]
y
[Equation]
t − y i (cid:105) )
=
[Equation]
(cid:88) t = (cid:104)
g c t
[Equation]
y t − x ∗ (cid:105) +
T + d − 1 (cid:88) t =
s τ t
+1 − 1 (cid:88)
[Equation]
≤
[Equation]
(cid:124)
[Equation]
(14) where the ﬁrst equality is due to (9), the second equality is due to i + d i − 1 = t for any i ∈ F t , thethird equality is due to (10), and the last equality is due to (7) and (8).
By combining (13) and (14), we have
[Equation]
Next, we proceed to bound terms A , B , and C deﬁned above.
Speciﬁcally, we ﬁrst establish thefollowing bound for the sum of terms A and C by carefully analyzing the distance (cid:107)
y τ t − y τ t (cid:48) (cid:107) 2 inthe term A and the distance (cid:107)
y τ t − y
i
(cid:107) 2 in the term C .
Lemma 1 Let y ∗ t = argmin
y ∈K F t − 1 ( y ) for any t ∈
[ T + 1] , where F t ( y ) is deﬁned in (11).
Suppose Assumption 1 and 2 hold, and there exist some constants γ > 0 and 0 < α ≤ 1 such that
F t − 1 ( y t ) − F t − 1 ( y ∗ t ) ≤ γ ( t + 2) − α for any t ∈
[ T + 1] , Algorithm 1 ensures
[Equation]
where A = (cid:80) Tt =
1 G (cid:107)
y τ t − y τ t (cid:48) (cid:107) 2 and C =
(cid:80) T + d − 1 t =
s
[Equation]
−
[Equation]
Note that Lemma 1 introduces an assumption about y t and F t − 1 ( y ) .
According to our Algorithm1, y t is actually generated by approximately minimizing F t − 1 ( y ) with a linear optimization step.
Therefore, by following the analysis of the original OFW [Hazan, 2016], we show that this assumptioncan be satisﬁed with γ = 8 D 2 and α = 1 / 2 .
Lemma 2 Let y ∗ t = argmin
y ∈K F t − 1 ( y ) for any t ∈
[ T +1] , where F t ( y ) is deﬁned in (11).
Under
Assumptions 1 and 2, for any t ∈
[ T + 1] , Algorithm 1 with η =
D √ 2 G
( T +2) 3 / 4 has
[Equation]
Then, by combining η =
D √ 2 G
( T +2) 3 / 4 with Lemmas 1 and 2, we have
[Equation]
Furthermore, by following the analysis of the original OFW [Hazan, 2016], we establish an upperbound for the term B .
Lemma 3 Under Assumptions 1 and 2, for any x ∗ ∈ K , Algorithm 1 with η = D [Equation]
2 G ( T +
[Equation] 3 / 4 ensures
T
[Equation]
Finally, by combining (15), (16), and Lemma 3, we complete this proof.
Experiments
In this section, we conduct simulation experiments to verify the performance of our delayed OFW forconvex losses and strongly convex losses.
All algorithms are implemented wtih Matlab R2017a andtested on a laptop with 2.4GHz CPU and 16GB memory.
Speciﬁcally, we consider a delayed OCO problem with T = 10000 total rounds over the decision set
K = (cid:8) x ∈ R 1000 (cid:12)(cid:12) (cid:107) x (cid:107) 1 ≤ 200 (cid:9) , which satisﬁes Assumption 2 with D = 400 due to (cid:107) x
− y (cid:107) 2 ≤(cid:107) x (cid:107) {+
+}
+ (cid:107) y (cid:107) 2 ≤ (cid:107) x (cid:107) 1 + (cid:107) y (cid:107) 1 ≤ 400 for any x , y ∈ K .
Inspired by Li et al.
[2019], in each roundt ∈
[ T ] , the adversary chooses the following loss function f t ( x ) = (cid:107) x (cid:107) 2 + (cid:104) b
t , x (cid:105) , where eachelement of b t ∈ R 1000 is independently and uniformly sampled from [ − 1 , 1] .
For any x ∈ K , since∇ f t ( x ) = 2 x + b t , it is easy to verify that (cid:107)∇ f t ( x ) (cid:107) 2 ≤ 2 (cid:107) x (cid:107) 2 + (cid:107) b t (cid:107) 2 ≤ 2 (cid:107) x (cid:107) 1 + (cid:107) b t (cid:107) 2 ≤400+ √ 1000 , which implies that this loss function satisﬁes Assumption 1 with G = 400+ √ 1000 .
Wealso notice that this loss function satisﬁes the deﬁnition of β -strongly convex function (i.e., Deﬁnition1) with β = 2 . Moreover, different values of the maximum delay d in the set { 1 , 51 , 101 , . .
. , 501 }are tried in our experiments. For each speciﬁc d , to simulate arbitrary delays, d t is independently anduniformly sampled from the set {(cid:100) 0 .
1 d (cid:101) , (cid:100) 0 .
1 d (cid:101) + 1 , (cid:100) 0 . 1 d (cid:101) + 2 , . . . , d } .
We compare our proposed algorithms against the combination of the technique in Joulani et al.
[2013]and OFW.
Our delayed OFW for convex losses and strongly convex losses are denoted as DOFW (i.e., Algorithm 1) and DOFW sc (i.e., Algorithm 2), respectively. Similarly, since the technique in Joulaniet al. [2013] is named as black-box online learning under delayed feedback (BOLD), we denotethe combination of BOLD with OFW for convex losses and strongly convex losses as BOLD-OFW
[Figure]
and BOLD-OFW sc , respectively. The parameters of all algorithms are set as their correspondingtheories suggest. Speciﬁcally, according to our Theorems 1 and 2, we need to set η =
[Figure]
for our DOFW and set β = 2 for our DOFW sc . According to Joulani et al. [2013], BOLD-OFWand BOLD-OFW sc will maintain several instances of OFW for convex and strongly convex losses,respectively.
Note that in the non-delayed case with d = 1 our delayed OFW actually reduces to theoriginal OFW. Therefore, our Theorems 1 and 2 can also be utilized to choose the parameters for eachinstance of OFW maintained in BOLD-OFW and BOLD-OFW sc . To be precise, in BOLD-OFW, weset η = D √ 2 G
( T/d +2) 3 / 4 for each instance of OFW for convex losses, since the total rounds of eachinstance is roughly T/d . In BOLD-OFW sc , we only need to set β = 2 for each instance of OFWfor strongly convex losses. Moreover, for our delayed OFW and each instance of OFW, the initialdecision is set to 1 / 5 , where 1 denotes the vector with each entry equal 1 . Fig.
1(a) shows the total loss of T rounds for each algorithm facing different values of the maximumdelay d . First, when d = 1 , the total loss of our DOFW is the same as that of BOLD-OFW andthe total loss of our DOFW sc is the same as that of BOLD-OFW sc , which is reasonable becausein this case DOFW and BOLD-OFW reduce to the original OFW for convex losses, and DOFW scand BOLD-OFW sc reduce to the original OFW for strongly convex losses. Second, for d ={ 51 , 101 , 151 , . . , 501 } , our DOFW and DOFW sc are better than BOLD-OFW and BOLD-OFW screspectively, which clearly veriﬁes the advantage of our algorithms in the delayed setting.
It is worthyto notice that d = 501 is larger than T 2 / 3 . Moreover, for our DOFW and DOFW sc , when d increasesfrom 1 to 501 , the growth of the total loss is very slow, which is consistent with the dependence ofour regret bounds on d . Fig. 1(b) shows the cumulative loss for each algorithm when d = 501 . In theﬁrst few iterations, all algorithms have the same cumulative loss, since none of them receives thedelayed gradient. Later, as the number of iterations increases, the cumulative loss of BOLD-OFWand BOLD-OFW sc increase much faster than that of our algorithms.
Conclusion and future work
In this paper, we propose delayed OFW for OCO with arbitrary delays.
For convex losses, we showthat it attains an O ( T 3 / 4 + dT 1 / 4 ) regret bound, which matches the O ( T 3 / 4 ) regret bound of OFW inthe non-delayed setting, as long as d does not exceed O ( √ T ) .
When losses are strongly convex, wefurther prove that it can attain an O ( T 2 / 3 + d log T ) regret bound, which matches the O ( T 2 / 3 ) regretbound of OFW in the non-delayed setting, as long as d does not exceed O ( T 2 / 3 / log T ) .
Simulationexperiments demonstrate the performance of delayed OFW in the delayed setting.
This paper only extends the classical OFW to the delayed setting.
In the future, we will investigatehow to develop delayed variants for other projection-free online algorithms.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]( b ) Did you describe the limitations of your work?
[Yes]( c)
Did you discuss any potential negative societal impacts of your work
[N/ A
This workis mostly theoretical and the societal impacts discussion is not applicable (d
Have you read the ethics review guidelines and ensured that your
paper conforms to
them?
[Yes]
If you are including theoretical results...
a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] We would like
to release our codes in the ﬁnal version of this work.
b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A] Only synthetic data are used in this work, which do not contain personally identiﬁable information and offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]