Original: O N THE D YNAMIC R EGRET OF O NLINE M ULTIPLE M IRROR D ESCENT
Revised: O N THE D YNAMIC R EGRET OF O NLINE M ULTIPLE M IRROR D ESCENT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study the problem of online convex optimization, where a learner makes sequential decisions to minimize an accumulation of strongly convex costs over time.
Revised: We study the problem of online convex optimization, where a learner makes sequential decisions to minimize an accumulation of strongly convex costs over time.

Original: The quality of decisions is given in terms of the dynamic regret, which measures the performance of the learner relative to a sequence of dynamic minimizers.
Revised: The quality of decisions is given in terms of the dynamic regret, which measures the performance of the learner relative to a sequence of dynamic minimizers.

Original: Prior works on gradient descent and mirror descent have shown that the dynamic regret can be upper bounded using the path length, which depend on the differences between successive minimizers, and an upper bound using the squared path length has also been shown when multiple gradient queries are allowed per round.
Revised: Prior works on gradient descent and mirror descent have shown that the dynamic regret can be upper bounded using the path length, which depend on the differences between successive minimizers, and an upper bound using the squared path length has also been shown when multiple gradient queries are allowed per round.

Original: However, they all require the cost functions to be Lipschitz continuous, which imposes a strong requirement especially when the cost functions are also strongly convex.
Revised: However, they all require the cost functions to be Lipschitz continuous, which imposes a strong requirement especially when the cost functions are also strongly convex.

Original: In this work, we consider Online Multiple Mirror Descent (OMMD), which is based on mirror descent but uses multiple mirror descent steps per online round.
Revised: In this work, we consider Online Multiple Mirror Descent (OMMD), which is based on mirror descent but uses multiple mirror descent steps per online round.

Original: Without requiring the cost functions to be Lipschitz continuous, we derive two upper bounds on the dynamic regret based on the path length and squared path length.
Revised: Without requiring the cost functions to be Lipschitz continuous, we derive two upper bounds on the dynamic regret based on the path length and squared path length.

Original: We further derive a third upper bound that relies on the gradient of cost functions, which can be much smaller than the path length or squared path length, especially when the cost functions are smooth but ﬂuctuate over time.
Revised: We further derive a third upper bound that relies on the gradient of cost functions, which can be much smaller than the path length or squared path length, especially when the cost functions are smooth but ﬂuctuate over time.

Original: Thus, we show that the dynamic regret of OMMD scales linearly with the minimum among the path length, squared path length, and sum squared gradients.
Revised: Thus, we show that the dynamic regret of OMMD scales linearly with the minimum among the path length, squared path length, and sum squared gradients.

Original: Our experimental results further show substantial improvement on the dynamic regret compared with existing alternatives.
Revised: Our experimental results further show substantial improvement on the dynamic regret compared with existing alternatives.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Online optimization refers to the design of sequential decisions where system parameters and cost functions vary with time.
Revised: Online optimization refers to the design of sequential decisions where system parameters and cost functions vary with time.

Original: It has applications to various classes of problems, such as object tracking (Shahrampour & Jadbabaie, 2017), networking (Shi et al., 2018), cloud computing (Lin et al., 2012), and classiﬁcation (Crammer et al., 2006).
Revised: It has applications to various classes of problems, such as object tracking (Shahrampour & Jadbabaie, 2017), networking (Shi et al., 2018), cloud computing (Lin et al., 2012), and classiﬁcation (Crammer et al., 2006).

Original: It is also an important tool in the development of algorithms for reinforcement learning (Yuan & Lamperski, 2017) and deep learning (Mnih et al., 2015).
Revised: It is also an important tool in the development of algorithms for reinforcement learning (Yuan & Lamperski, 2017) and deep learning (Mnih et al., 2015).

Original: In this work, we consider online convex optimization, which can be formulated as a discrete-time sequential learning process as follows.
Revised: In this work, we consider online convex optimization, which can be formulated as a discrete-time sequential learning process as follows.

Original: At each round t , the learner ﬁrst makes a decision x t ∈ X , where X is a convex set representing the solution space.
Revised: At each round t , the learner ﬁrst makes a decision x t ∈ X , where X is a convex set representing the solution space.

Original: The learner then receives a convex cost function f t ( x ) :
Revised: The learner then receives a convex cost function f t ( x ) :

Original: X → R and suffers the corresponding cost of f t ( x t ) associated with the submitted decision.
Revised: X → R and suffers the corresponding cost of f t ( x t ) associated with the submitted decision.

Original: The goal of the online learner is to minimize the total accrued cost over a ﬁnite number of rounds, denoted by T .
Revised: The goal of the online learner is to minimize the total accrued cost over a ﬁnite number of rounds, denoted by T .

Original: For performance evaluation, prior studies on online learning often focus on the static regret, deﬁned as the difference between the learner’s accumulated cost and that of an optimal ﬁxed ofﬂine decision, which is made in hindsight with knowledge of f t ( · ) for all t :
Revised: For performance evaluation, prior studies on online learning often focus on the static regret, deﬁned as the difference between the learner’s accumulated cost and that of an optimal ﬁxed ofﬂine decision, which is made in hindsight with knowledge of f t ( · ) for all t :

Original: [Equation]
Revised: [Equation]

Original: A successful online algorithm closes the gap between the online decisions and the ofﬂine counterpart when normalized by T , i.e., sustaining sublinear static regret in T .
Revised: A successful online algorithm closes the gap between the online decisions and the ofﬂine counterpart when normalized by T , i.e., sustaining sublinear static regret in T .

Original: In the literature, there are various online algorithms (Zinkevich, 2003; Cesa-Bianchi & Lugosi, 2006; Hazan et al., 2006; Duchi et al., 2010; Shalev-Shwartz, 2012) that guarantee a sublinear bound on the static regret.
Revised: In the literature, there are various online algorithms (Zinkevich, 2003; Cesa-Bianchi & Lugosi, 2006; Hazan et al., 2006; Duchi et al., 2010; Shalev-Shwartz, 2012) that guarantee a sublinear bound on the static regret.

Original: However, algorithms that guarantee performance close to that of a static decision may still perform poorly in dynamic settings.
Revised: However, algorithms that guarantee performance close to that of a static decision may still perform poorly in dynamic settings.

Original: Consequently, the static regret fails to accurately reﬂect the quality of decisions in many practical scenarios.
Revised: Consequently, the static regret fails to accurately reﬂect the quality of decisions in many practical scenarios.

Original: Therefore, the dynamic regret has become a popular metric in recent works (Zinkevich, 2003; Besbes et al., 2015; Hall & Willett, 2015; Jadbabaie et al., 2015;
Revised: Therefore, the dynamic regret has become a popular metric in recent works (Besbes et al., 2015; Mokhtari et al., 2016;

Original: Mokhtari et al., 2016; Yang et al., 2016; Zhang et al., 2017), which allows a dynamic sequence of comparison targets and is deﬁned by
Revised: Yang et al., 2016; Zhang et al., 2017), which allows a dynamic sequence of comparison targets and is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where x ∗ t = argmin x ∈X f t ( x ) is a minimizer of the cost at round t .
Revised: where x ∗ t = argmin x ∈X f t ( x ) is a minimizer of the cost at round t .

Original: It is well-known that the online optimization problem may be intractable in a dynamic setting, due to arbitrary ﬂuctuation in the cost functions.
Revised: It is well-known that the online optimization problem may be intractable in a dynamic setting, due to arbitrary ﬂuctuation in the cost functions.

Original: Hence, achieving a sublinear bound on the dynamic regret may be impossible.
Revised: Hence, achieving a sublinear bound on the dynamic regret may be impossible.

Original: However, it is possible to upper bound the dynamic regret in terms of certain regularity measures.
Revised: However, it is possible to upper bound the dynamic regret in terms of certain regularity measures.

Original: One of the measures to represent regularity is the path length , deﬁned by
Revised: One of the measures to represent regularity is the path length , deﬁned by

Original: [Equation]
Revised: [Equation]

Original: which illustrates the accumulative variation in the minimizer sequence.
Revised: which illustrates the accumulative variation in the minimizer sequence.

Original: For instance, the dynamic regret of online gradient descent for convex cost functions can be bounded by O ( √ T (1 + C T ))
Revised: For instance, the dynamic regret of online gradient descent for convex cost functions can be bounded by O ( √ T (1 + C [Equation]

Original:  (Zinkevich, 2003).
Revised: )) (Zinkevich, 2003).

Original: For strongly convex functions, the dynamic regret of online gradient descent can be reduced to O ( C T ) (Mokhtari et al., 2016).
Revised: For strongly convex functions, the dynamic regret of online gradient descent can be reduced to O ( C T ) (Mokhtari et al., 2016).

Original: When the cost functions are smooth and strongly convex, by allowing the learner to make multiple queries to the gradient of the cost functions, the regret bound can be further improved to O (min( C T , S T )) , where S T represents the squared path length , deﬁned by
Revised: When the cost functions are smooth and strongly convex, by allowing the learner to make multiple queries to the gradient of the cost functions, the regret bound can be further improved to O (min( C T , S T )) , where S T represents the squared path length , deﬁned by

Original: [Equation]
Revised: [Equation]

Original: which can be smaller than the path length when the distance between successive minimizers is small.
Revised: which can be smaller than the path length when the distance between successive minimizers is small.

Original: All the aforementioned studies require the cost functions to be Lipschitz continuous.
Revised: All the aforementioned studies require the cost functions to be Lipschitz continuous.

Original: However, there are many commonly used cost functions, e.g., the quadratic function that do not meet the Lipschitz condition.
Revised: However, there are many commonly used cost functions, e.g., the quadratic function that do not meet the Lipschitz condition.

Original: In addition, the above works rely on measuring distances using Euclidean norms, which hinders the projection step in gradient descent update for some constraint sets, e.g., probability simplex (Duchi, 2018).
Revised: In addition, the above works rely on measuring distances using Euclidean norms, which hinders the projection step in gradient descent update for some constraint sets, e.g., probability simplex (Duchi, 2018).

Original: Besides gradient descent, mirror descent is another well-known technique of online convex optimization (Hall & Willett, 2015; Jadbabaie et al., 2015).
Revised: Besides gradient descent, mirror descent is another well-known technique of online convex optimization (Hall & Willett, 2015; Jadbabaie et al., 2015).

Original: Mirror descent uses the Bregman divergence, which generalizes the Euclidean norm used in the projection step of gradient descent, thus acquiring expanded applicability to a broader range of problems.
Revised: Mirror descent uses the Bregman divergence, which generalizes the Euclidean norm used in the projection step of gradient descent, thus acquiring expanded applicability to a broader range of problems.

Original: In addition, the Bregman divergence is only mildly dependent on the dimension of decision variables (Beck & Teboulle, 2003; Nemirovsky &
Revised: In addition, the Bregman divergence is only mildly dependent on the dimension of decision variables (Beck & Teboulle, 2003; Nemirovsky &

Original: Yudin, 1983), so that mirror descent is optimal among ﬁrst-order methods when the decision variables have high dimensions (Duchi et al., 2010).
Revised: Yudin, 1983), so that mirror descent is optimal among ﬁrst-order methods when the decision variables have high dimensions (Duchi et al., 2010).

Original: In this work we focus on the mirror descent approach.
Revised: In this work we focus on the mirror descent approach.

Original: In previous works on online mirror descent, the learner queries the gradient of each cost function only once, and performs one step of mirror descent to update its decision (Hall & Willett, 2015;
Revised: In previous works on online mirror descent, the learner queries the gradient of each cost function only once, and performs one step of mirror descent to update its decision (Hall & Willett, 2015;

Original: Shahrampour & Jadbabaie, 2017).
Revised: Shahrampour & Jadbabaie, 2017).

Original: In this case, the dynamic regret has an upper bound of order
Revised: In this case, the dynamic regret has an upper bound of order

Original: O ( √ T (1 + C T )) , which is the same as that of online gradient descent in (Zinkevich, 2003).
Revised: O ( √ T (1 + C T )) , which is the same as that of online gradient descent in (Zinkevich, 2003).

Original: In this work, we investigate whether it is possible to improve the dynamic regret when the learner performs multiple mirror descent steps in each online round, while relaxing the Lipschitz continuity condition on the cost functions.
Revised: In this work, we investigate whether it is possible to improve the dynamic regret when the learner performs multiple mirror descent steps in each online round, while relaxing the Lipschitz continuity condition on the cost functions.

Original: To this end, we analyze the performance of the Online Multiple Mirror Descent (OMMD) algorithm, which uses multiple steps of mirror descent per online round.
Revised: To this end, we analyze the performance of the Online Multiple Mirror Descent (OMMD) algorithm, which uses multiple steps of mirror descent per online round.

Original: When the cost functions are smooth and strongly convex, we show that the upper bound on the dynamic regret can be reduced from
Revised: When the cost functions are smooth and strongly convex, we show that the upper bound on the dynamic regret can be reduced from

Original: O ( √ T (1 + C T )) to O (min( C T , S T , G T )) , where G T represent the sum squared gradients , i.e.,
Revised: O ( √ T (1 + C T )) to O (min( C T , S T , G T )) , where G T represent the sum squared gradients , i.e.,

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) .
Revised: where (cid:107) .

Original: (cid:107) ∗ denotes the dual norm.
Revised: (cid:107) ∗ denotes the dual norm.

Original: The sum squared gradients G T can be smaller than both the path length and squared path length, especially when the cost functions ﬂuctuate drastically over time.
Revised: The sum squared gradients G T can be smaller than both the path length and squared path length, especially when the cost functions ﬂuctuate drastically over time.

Original: In contrast to the aforementioned works, our analysis does not require the cost functions to be Lipschitz continuous.
Revised: In contrast to the aforementioned works, our analysis does not require the cost functions to be Lipschitz continuous.

Original: Furthermore, our numerical experiments suggest substantially reduced dynamic regret compared with the best known alternatives, including single-step dynamic mirror descent (Hall &
Revised: Furthermore, our numerical experiments suggest substantially reduced dynamic regret compared with the best known alternatives, including single-step dynamic mirror descent (Hall &

Original: Willett, 2015), online multiple gradient descent (Zhang et al., 2017), and online gradient descent
Revised: Willett, 2015), online multiple gradient descent (Zhang et al., 2017), and online gradient descent

Original: Zinkevich (2003).
Revised: Zinkevich (2003).

Original:  O NLINE M ULTIPLE M IRROR D ESCENT
Revised:  O NLINE M ULTIPLE M IRROR D ESCENT

Original: In this section, we describe OMMD and discuss how the learner can improve the dynamic regret by performing multiple mirror descent steps per round.
Revised: In this section, we describe OMMD and discuss how the learner can improve the dynamic regret by performing multiple mirror descent steps per round.

Original: Before delving into the details, we proceed by stating several deﬁnitions and standard assumptions.
Revised: Before delving into the details, we proceed by stating several deﬁnitions and standard assumptions.

Original: 2. P RELIMINARIES
Revised: 2. P RELIMINARIES

Original: Deﬁnition 1 : The Bregman divergence with respect to the regularization function r ( · ) is deﬁned as
Revised: Deﬁnition 1 : The Bregman divergence with respect to the regularization function r ( · ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The Bregman divergence is a general distance-measuring function, which contains the Euclidean norm and the Kullback-Leibler divergence as two special cases.
Revised: The Bregman divergence is a general distance-measuring function, which contains the Euclidean norm and the Kullback-Leibler divergence as two special cases.

Original: Using the Bregman divergence, a generalized deﬁnition of strong convexity is given in (ShalevShwartz & Singer, 2007).
Revised: Using the Bregman divergence, a generalized deﬁnition of strong convexity is given in (ShalevShwartz & Singer, 2007).

Original: Deﬁnition 2 : A convex function f ( · ) is λ -strongly convex with respect to a convex and differentiable function r ( · ) if
Revised: Deﬁnition 2 : A convex function f ( · ) is λ -strongly convex with respect to a convex and differentiable function r ( · ) if

Original: [Equation]
Revised: [Equation]

Original: Following many prior studies on mirror descent, we assume that the cost functions are λ -strongly convex, where the above generalized strong convexity deﬁnition is used.
Revised: Following many prior studies on mirror descent, we assume that the cost functions are λ -strongly convex, where the above generalized strong convexity deﬁnition is used.

Original: We further assume that the cost functions are L -smooth, and the regularization function r ( · ) is L r -smooth and 1 -strongly convex with respect to some norm (refer to App.
Revised: We further assume that the cost functions are L -smooth, and the regularization function r ( · ) is L r -smooth and 1 -strongly convex with respect to some norm (refer to App.

Original: A for deﬁnitions).
Revised: A for deﬁnitions).

Original: We note that these are standard assumptions commonly used in the literature after the group of studies began by (Hazan et al., 2006;
Revised: We note that these are standard assumptions commonly used in the literature after the group of studies began by (Hazan et al., 2006;

Original: Shalev-Shwartz & Singer, 2007), to provide stronger regret bounds by constraining the curvature of cost functions.
Revised: Shalev-Shwartz & Singer, 2007), to provide stronger regret bounds by constraining the curvature of cost functions.

Original: We further make a standard assumption that the Bregman divergence is Lipschitz continuous as follows:
Revised: We further make a standard assumption that the Bregman divergence is Lipschitz continuous as follows:

Original: [Equation]
Revised: [Equation]

Original: where K is a positive constant.
Revised: where K is a positive constant.

Original: We note that this condition is much milder than the condition of
Revised: We note that this condition is much milder than the condition of

Original: Lipschitz continuous cost functions required in (Zhang et al., 2017; Mokhtari et al., 2016; Hall & 
Revised: Lipschitz continuous cost functions required in (Zhang et al., 2017; Mokhtari et al., 2016; Hall & Willett, 2015). There is a notable weakness in such bounds.

Original: Willett, 2015), since it can be easily satisﬁed with a proper choice of  Lipschitz continuous function r ( · ) on the compact set X .
Revised: Since the sequence of cost functions are revealed to the learner, the learner has no control over it. If these cost functions happen to not meet the Lipschitz condition, earlier analyses that require this condition become inapplicable. In this work, we do not require the cost functions to be Lipschitz continuous. Instead, we move the Lipschitz continuity condition from the cost functions to the Bregman divergence to broaden the application of our work.

Original: In practical problems, one often has the freedom to design r ( · ) but not the cost functions.
Revised: The main beneﬁt of this is that the regularization function and the corresponding Bregman divergence is within the control of the learner. The learner can carefully design this regularization function to satisfy the Lipschitz continuity of the associated Bregman divergence with a small factor. For example, in the particular case of the KL divergence, which is obtained by the choosing negative entropy as the regularization function, on the set X = { x | (cid:80) di =1 x i = 1; x i ≥ 1 D } , the constant K is of O (log D ) .

Original: Examples of many widely used Bregman divergences that satisfy this condition are given in (Bauschke & Borwein, 2001).
Revised: Other examples of many widely used Bregman divergences that satisfy this condition are given in (Bauschke & Borwein, 2001).

Original: 2. O NLINE C ONVEX O PTIMIZATION WITH OMMD
Revised: O NLINE C ONVEX O PTIMIZATION WITH OMMD

Original: We consider online optimization over a ﬁnite number of rounds, denoted by T .
Revised: We consider online optimization over a ﬁnite number of rounds, denoted by T .

Original: At the beginning of every round t , the learner submits a decision represented by x t , which is taken from a convex and compact set X .
Revised: At the beginning of every round t , the learner submits a decision represented by x t , which is taken from a convex and compact set X .

Original: Then, an adversary selects a function f t ( · ) and the learner suffers the corresponding cost f t ( x t ) .
Revised: Then, an adversary selects a function f t ( · ) and the learner suffers the corresponding cost f t ( x t ) .

Original: The learner then updates its decision in the next round.
Revised: The learner then updates its decision in the next round.

Original: With standard mirror descent, this is given by
Revised: With standard mirror descent, this is given by

Original: [Equation]
Revised: [Equation]

Original: where α is a ﬁxed step size, and D r ( · , · ) is the Bregman divergence corresponding to the regularization function r ( · ) .
Revised: where α is a ﬁxed step size, and D r ( · , · ) is the Bregman divergence corresponding to the regularization function r ( · ) .

Original: The update in equation 4 suggests that the learner aims to stay close to the current decision x t as measured by the Bregman divergence, while taking a step in a direction close to the negative gradient to reduce the current cost at round t .
Revised: The update in equation 4 suggests that the learner aims to stay close to the current decision x t as measured by the Bregman divergence, while taking a step in a direction close to the negative gradient to reduce the current cost at round t .

Original: OMMD uses mirror descent in its core as the optimization workhorse.
Revised: OMMD uses mirror descent in its core as the optimization workhorse.

Original: However, in contrast to classical online optimization methods, where the learner queries the gradient of each cost function only once, OMMD is designed to take advantage of the curvature of cost functions by allowing the learner to make multiple queries to the gradient in each round.
Revised: However, in contrast to classical online optimization methods, where the learner queries the gradient of each cost function only once, OMMD is designed to take advantage of the curvature of cost functions by allowing the learner to make multiple queries to the gradient in each round.

Original: This is especially important when the successive cost functions have similar curvatures.
Revised: This is especially important when the successive cost functions have similar curvatures.

Original: In particular, in order to track x ∗ t +1 the learner needs to access the gradient of the cost function, i.e., ∇ f t +1 ( · ) .
Revised: In particular, in order to track x ∗ t +1 the learner needs to access the gradient of the cost function, i.e., ∇ f t +1 ( · ) .

Original: Unfortunately, this information is not available until the end of round t + 1 .
Revised: Unfortunately, this information is not available until the end of round t + 1 .

Original: However, if the successive functions have similar curvatures, the gradient of f t ( · ) is a reasonably accurate estimate for the gradient of f t +1 ( · ) .
Revised: However, if the successive functions have similar curvatures, the gradient of f t ( · ) is a reasonably accurate estimate for the gradient of f t +1 ( · ) .

Original: In this case, every time that the learner queries the gradient of f t ( · ) , it ﬁnds a point that is likely to be closer to the minimizer of f t +1 ( · ) .
Revised: In this case, every time that the learner queries the gradient of f t ( · ) , it ﬁnds a point that is likely to be closer to the minimizer of f t +1 ( · ) .

Original: Hence, it may beneﬁt the learner to perform multiple mirror descent steps in each round.
Revised: Hence, it may beneﬁt the learner to perform multiple mirror descent steps in each round.

Original: Thus, the learner generates a series of decisions, represented by y 1 t , y 2 t , . . .
Revised: Thus, the learner generates a series of decisions, represented by y 1 t , y 2 t , . . .

Original: , y M +1 t , via the following updates:
Revised: , y M +1 t , via the following updates:

Original: [Equation]
Revised: [Equation]

Original: Then, by setting x t +1 =
Revised: Then, by setting x t +1 =

Original: y M +1 t , the learner proceeds to the next round, and the procedure continues.
Revised: y M +1 t , the learner proceeds to the next round, and the procedure continues.

Original: Note that M is independent of T .
Revised: Note that M is independent of T .

Original: Applying multiple steps of mirror descent can reveal more information about the sequence of minimizers.
Revised: Applying multiple steps of mirror descent can reveal more information about the sequence of minimizers.

Original: It can reduce the dynamic regret, but only if the series of decisions in equation 5 helps decrease the distance to the minimizer x ∗ t +1 .
Revised: It can reduce the dynamic regret, but only if the series of decisions in equation 5 helps decrease the distance to the minimizer x ∗ t +1 .

Original: Therefore, quantifying the beneﬁt of OMMD over standard mirror descent requires careful analysis on the impact of the ﬂuctuation of f t ( · ) over time.
Revised: Therefore, quantifying the beneﬁt of OMMD over standard mirror descent requires careful analysis on the impact of the ﬂuctuation of f t ( · ) over time.

Original: To this end, we provide an analysis to bound the dynamic regret of OMMD in the next section.
Revised: To this end, we provide an analysis to bound the dynamic regret of OMMD in the next section.

Original:  T HEORETICAL R ESULTS
Revised:  T HEORETICAL R ESULTS

Original: The following lemma paves the way for the proposed analysis on the dynamic regret of OMMD.
Revised: The following lemma paves the way for the proposed analysis on the dynamic regret of OMMD.

Original: It bounds the distance of the learner’s future decision from the current optimal solution, after a single step of mirror descent.
Revised: It bounds the distance of the learner’s future decision from the current optimal solution, after a single step of mirror descent.

Original: Lemma 1 Assume that f t ( · ) is λ -strongly convex with respect to a differentiable function r ( · ) , and is L -smooth.
Revised: Lemma 1 Assume that f t ( · ) is λ -strongly convex with respect to a differentiable function r ( · ) , and is L -smooth.

Original: Single-step mirror descent with a ﬁxed step size α ≤ 1 L guarantees the following:
Revised: Single-step mirror descent with a ﬁxed step size α ≤ 1 L guarantees the following:

Original: [Equation]
Revised: [Equation]

Original: where x ∗ t is the unique minimizer of f t ( · ) , and β = 1 − 2 αλ 1+ αλ .
Revised: where x ∗ t is the unique minimizer of f t ( · ) , and β = 1 − 2 αλ 1+ αλ .

Original: Lemma 1 is proved in App.
Revised: Lemma 1 is proved in App.

Original: B in the supplementary material.
Revised: B in the supplementary material.

Original: Remark 1.
Revised: Remark 1.

Original: Lemma 1 states that a mirror descent step reduces the distance (measured by the Bregman divergence) of the learner’s decisions to the current minimizer.
Revised: Lemma 1 states that a mirror descent step reduces the distance (measured by the Bregman divergence) of the learner’s decisions to the current minimizer.

Original: This generalizes the results in (Mokhtari et al., 2016; Zhang et al., 2017), where similar bounds were derived for online gradient descent when the distance was measured in Euclidean norms.
Revised: This generalizes the results in (Mokhtari et al., 2016; Zhang et al., 2017), where similar bounds were derived for online gradient descent when the distance was measured in Euclidean norms.

Original: In particular, those results correspond to the special choice of r ( x ) = (cid:107) x (cid:107) 22 , which reduces the Bregman divergence to Euclidean distance, i.e.,
Revised: In particular, those results correspond to the special choice of r ( x ) = (cid:107) x (cid:107) 22 , which reduces the Bregman divergence to Euclidean distance, i.e.,

Original: D r ( x, y ) =
Revised: D r ( x, y ) =

Original: (cid:107) x − y (cid:107) 2 .
Revised: (cid:107) x − y (cid:107) 2 .

Original: Lemma 1 indicates that the distance between the next decision x t +1 and the minimizer x ∗ t is strictly smaller than the distance between the current decision x t and the minimizer at round t .
Revised: Lemma 1 indicates that the distance between the next decision x t +1 and the minimizer x ∗ t is strictly smaller than the distance between the current decision x t and the minimizer at round t .

Original: This implies that if the minimizers of the functions f t ( · ) and f t +1 ( · ) , which are x ∗ t and x ∗ t +1 respectively, are not far from each other, applying mirror descent multiple times enables the online learner to more accurately track the sequence of optimal solutions x ∗ t .
Revised: This implies that if the minimizers of the functions f t ( · ) and f t +1 ( · ) , which are x ∗ t and x ∗ t +1 respectively, are not far from each other, applying mirror descent multiple times enables the online learner to more accurately track the sequence of optimal solutions x ∗ t .

Original: The succeeding theorems provide three separate upper bounds on the dynamic regret of OMMD, based on path length C T (as deﬁned in equation 1), squared path length S T (as deﬁned in equation 2), and sum squared gradients (as deﬁned in equation 3).
Revised: The succeeding theorems provide three separate upper bounds on the dynamic regret of OMMD, based on path length C T (as deﬁned in equation 1), squared path length S T (as deﬁned in equation 2), and sum squared gradients (as deﬁned in equation 3).

Original: Theorem 2 Assume that r ( · ) is L r -smooth and 1 -strongly convex with respect to some norm (cid:107)·(cid:107) and that the cost functions are L -smooth and λ -strongly convex with respect to r ( · ) .
Revised: Theorem 2 Assume that r ( · ) is L r -smooth and 1 -strongly convex with respect to some norm (cid:107)·(cid:107) and that the cost functions are L -smooth and λ -strongly convex with respect to r ( · ) .

Original: Let x t be the sequence of decisions generated by OMMD with a ﬁxed step size 12 λ <
Revised: Let x t be the sequence of decisions generated by OMMD with a ﬁxed step size 12 λ <

Original: α ≤ 1 L and M ≥ (cid:100) (cid:0) 12 + 12 αλ (cid:1) log L r (cid:101) mirror descent steps per round.
Revised: α ≤ 1 L and M ≥ (cid:100) (cid:0) 12 + 12 αλ (cid:1) log L r (cid:101) mirror descent steps per round.

Original: The dynamic regret satisﬁes the following bound:
Revised: The dynamic regret satisﬁes the following bound:

Original: [Equation]
Revised: [Equation]

Original: where β is the shrinking factor derived in Lemma 1, and K is the Lipschitz constant associated with
Revised: where β is the shrinking factor derived in Lemma 1, and K is the Lipschitz constant associated with

Original: D r ( · , · ) .
Revised: D r ( · , · ) .

Original: The proof of Theorem 2 is given in App.
Revised: The proof of Theorem 2 is given in App.

Original: C in the supplementary material.
Revised: C in the supplementary material.

Original: Remark 2.
Revised: Remark 2.

Original: It has been shown in (Hall & Willett, 2015) that single-step mirror descent guarantees an upper bound of O ( √ T (1 + C T )) on the dynamic regret for convex cost functions.
Revised: It has been shown in (Hall & Willett, 2015) that single-step mirror descent guarantees an upper bound of O ( √ T (1 + C T )) on the dynamic regret for convex cost functions.

Original: With that bound, a sublinear path length is not sufﬁcient to guarantee sublinear dynamic regret.
Revised: With that bound, a sublinear path length is not sufﬁcient to guarantee sublinear dynamic regret.

Original: In contrast, Theoremimplies that OMMD reduces the upper bound to O ( C T ) when the cost functions are strongly convex and smooth, which implies that a sublinear path length is sufﬁcient to yield sublinear dynamic regret.
Revised: In contrast, Theoremimplies that OMMD reduces the upper bound to O ( C T ) when the cost functions are strongly convex and smooth, which implies that a sublinear path length is sufﬁcient to yield sublinear dynamic regret.

Original: Remark 3.
Revised: Remark 3.

Original: The range of M where the bound in Theorem 2 holds is usually wide.
Revised: The range of M where the bound in Theorem 2 holds is usually wide.

Original: For example, it is
Revised: For example, it is

Original: M ≥ 3 and M ≥ 5 for the two experiments shown in Section 4.
Revised: M ≥ 3 and M ≥ 5 for the two experiments shown in Section 4.

Original: Theorem 3 Under the same convexity and smoothness conditions stated in Theorem 2, let x t be the sequence of decisions generated by OMMD with a ﬁxed step size α ≤ 1 L and M ≥ (cid:100) (cid:0) 
+ 12 αλ [Equation]
Revised: Theorem 3 Under the same convexity and smoothness conditions stated in Theorem 2, let x t be the sequence of decisions generated by OMMD with a ﬁxed step size α ≤ 1 L and M ≥ (cid:100) (cid:0)

Original:  log 2 L r (cid:101) mirror descent steps per round.
Revised: + 12 αλ (cid:1) log 2 L r (cid:101) mirror descent steps per round.

Original: For any arbitrary positive constant θ , the dynamic regret is upper bounded by
Revised: For any arbitrary positive constant θ , the dynamic regret is upper bounded by

Original: [Equation]
Revised: [Equation]

Original: Theorem 3 is proved in App.
Revised: Theorem 3 is proved in App.

Original: D in the supplementary material.
Revised: D in the supplementary material.

Original: Since the gradient at x ∗ t is zero if x ∗ t is in the relative interior of the feasibility set X , i.e., (cid:107)∇ f t ( x ∗ t ) (cid:107) = 0 , the above theorem can be simpliﬁed to the following corollary.
Revised: Since the gradient at x ∗ t is zero if x ∗ t is in the relative interior of the feasibility set X , i.e., (cid:107)∇ f t ( x ∗ t ) (cid:107) = 0 , the above theorem can be simpliﬁed to the following corollary.

Original: Corollary 4 If x ∗ t belongs to the relative interior of the feasibility set X for all t , the dynamic regret bound in Theorem 3 is of order O ( S T ) .
Revised: Corollary 4 If x ∗ t belongs to the relative interior of the feasibility set X for all t , the dynamic regret bound in Theorem 3 is of order O ( S T ) .

Original: When the cost functions drift slowly, the distances between successive minimizers are small.
Revised: When the cost functions drift slowly, the distances between successive minimizers are small.

Original: Hence, the squared path length S T , which relies on the square of those distances, can be signiﬁcantly smaller than the path length C T .
Revised: Hence, the squared path length S T , which relies on the square of those distances, can be signiﬁcantly smaller than the path length C T .

Original: In this case, Theorem 3 and Corollary 4 can provide a tighter regret bound than Theorem 2.
Revised: In this case, Theorem 3 and Corollary 4 can provide a tighter regret bound than Theorem 2.

Original: Theorem 5 Under the same convexity and smoothness conditions stated in Theorem 2, let x t be the sequence of decisions generated by OMMD with a ﬁxed step size α > 1 2 λ .
Revised: Theorem 5 Under the same convexity and smoothness conditions stated in Theorem 2, let x t be the sequence of decisions generated by OMMD with a ﬁxed step size α > 12 λ .

Original: The following bound holds on the dynamic regret:
Revised: The following bound holds on the dynamic regret:

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem 5 is given in App.
Revised: The proof of Theorem 5 is given in App.

Original: E in the supplementary material.
Revised: E in the supplementary material.

Original: Remark 4.
Revised: Remark 4.

Original: Interestingly, Theorem 5 implies that sublinear dynamic regret can be achieved when the gradient of the cost functions shrink over time.
Revised: Interestingly, Theorem 5 implies that sublinear dynamic regret can be achieved when the gradient of the cost functions shrink over time.

Original: For instance, if (cid:107)∇ f t ( x ) (cid:107) ∗ =
Revised: For instance, if (cid:107)∇ f t ( x ) (cid:107) ∗ =

Original: O (1 /t γ ) for some γ > 0 , Theorem 5 guarantees O ( T 1 − 2 γ ) dynamic regret.
Revised: O (1 /t γ ) for some γ > 0 , Theorem 5 guarantees O ( T 1 − 2 γ ) dynamic regret.

Original: This is especially important when the cost functions decrease while the minimizers ﬂuctuate.
Revised: This is especially important when the cost functions decrease while the minimizers ﬂuctuate.

Original: In this scenario, the path length C T and squared path length
Revised: In this scenario, the path length C T and squared path length

Original: S T may grow linearly, whereas diminishing gradients ensure sublinear G T .
Revised: S T may grow linearly, whereas diminishing gradients ensure sublinear G T .

Original: Theorem 2, Corollary 4, and Theorem 5, respectively, state that the dynamic regret of OMMD is upper bounded linearly by path length C T , squared path length S T , and sum squared gradients G T .
Revised: Theorem 2, Corollary 4, and Theorem 5, respectively, state that the dynamic regret of OMMD is upper bounded linearly by path length C T , squared path length S T , and sum squared gradients G T .

Original: This immediately leads to the following result.
Revised: This immediately leads to the following result.

Original: Corollary 6 Under the same convexity and smoothness conditions stated in Theorem 2, the dynamic regret of OMMD with suitably chosen α and M has an upper bound of O (min( C T , S T , G T )) .
Revised: Corollary 6 Under the same convexity and smoothness conditions stated in Theorem 2, the dynamic regret of OMMD with suitably chosen α and M has an upper bound of O (min( C T , S T , G T )) .

Original: Remark 5.
Revised: Remark 5.

Original: We note that (Mokhtari et al., 2016) and (Zhang et al., 2017) provide upper bounds of O ( C T ) and O (min( C T , S T )) , respectively, on the dynamic regret of online gradient descent with single and multiple gradient queries, while (Hall & Willett, 2015) presents an upper bound of
Revised: We note that (Mokhtari et al., 2016) and (Zhang et al., 2017) provide upper bounds of O ( C T ) and O (min( C T , S T )) , respectively, on the dynamic regret of online gradient descent with single and multiple gradient queries, while (Hall & Willett, 2015) presents an upper bound of

Original: O ( √ T (1 + C T )) on the dynamic regret of online single-step mirror descent.
Revised: O ( √ T (1 + C T )) on the dynamic regret of online single-step mirror descent.

Original: Corollary 6 shows that
Revised: Corollary 6 shows that

Original: OMMD can improve the dynamic regret bound to O (min( C T , S T , G T )) .
Revised: OMMD can improve the dynamic regret bound to O (min( C T , S T , G T )) .

Original: Furthermore, in contrast to these studies, our analysis does not require the cost functions to be Lipschitz continuous.
Revised: Furthermore, in contrast to these studies, our analysis does not require the cost functions to be Lipschitz continuous.

Original: The quantities C t , S T , and G T represent distinct aspects of an online learning problem and are not generally comparable.
Revised: The quantities C t , S T , and G T represent distinct aspects of an online learning problem and are not generally comparable.

Original: The following example demonstrates the beneﬁt of having multiple upper bounds and taking their minimum.
Revised: The following example demonstrates the beneﬁt of having multiple upper bounds and taking their minimum.

Original: Consider a sequence of quadratic programming problems of the form f t ( x ) = (cid:107)
Revised: Consider a sequence of quadratic programming problems of the form f t ( x ) = (cid:107)

Original: A t x − b t (cid:107) 2 over the d -dimensional probability simplex.
Revised: A t x − b t (cid:107) 2 over the d -dimensional probability simplex.

Original: Assume that for any t ≥ 1 , we have the parameter sequence of
Revised: Assume that for any t ≥ 1 , we have the parameter sequence of

Original: A t = (cid:26) diag ( 1 t p 1 , 0 , 0 , . . . , 0) , if t is odd diag (0 , 1 t p 1 , 0 , . . . , 0) , if t is even , and
Revised: [Equation]

Original:  b = [ 1 t p 2 , 1 t p 2 , . . .
Revised: and b = [ 1 t p 2 , 1 t p 2 , . . .

Original: , 1 t p 2 ] (cid:48) , where p 1 and p 2 are positive constants such that p 2 ≤ p 1 .
Revised: , 1 t p 2 ] (cid:48) , where p 1 and p 2 are positive constants such that p 2 ≤ p 1 .

Original: In this setting, we observe that C T
Revised: In this setting, we observe that C T

Original: = O ( T ) and G T
Revised: = O ( T ) and G T

Original: = O ( T 1 − p 1 − p 2 ) .
Revised: = O ( T 1 − p 1 − p 2 ) .

Original: Thus, G T can be considerably smaller than
Revised: Thus, G T can be considerably smaller than

Original: C T .
Revised: C T .

Original: On the other hand, it is also possible that C T is smaller than G T in other cases.
Revised: On the other hand, it is also possible that C T is smaller than G T in other cases.

Original: For example, let A t = diag (1 / 2 , 0 , . . .
Revised: For example, let A t = diag (1 / 2 , 0 , . . .

Original: , 0) on odd rounds and A t = diag (0 , − 1 / 2 , . .
Revised: , 0) on odd rounds and A t = diag (0 , − 1 / 2 , . .

Original: , − 1 / 2) , and b t be the unity vector for all t .
Revised: , − 1 / 2) , and b t be the unity vector for all t .

Original: In this case, we observe that C T
Revised: In this case, we observe that C T

Original: = O (1) , while the sum gradient scales linearly with time, i.e., G T
Revised: = O (1) , while the sum gradient scales linearly with time, i.e., G T

Original: = O ( T ) .
Revised: = O ( T ) .

Original: Thus, neither C T nor G T alone can provide a small regret bound for all cases.
Revised: Thus, neither C T nor G T alone can provide a small regret bound for all cases.

Original: Similar examples can be found in comparison between S T and G T but are omitted for brevity.
Revised: Similar examples can be found in comparison between S T and G T but are omitted for brevity.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We investigate the performance of OMMD via numerical experiments in two different learning scenarios (with further experiments presented in App.
Revised: We investigate the performance of OMMD via numerical experiments in two different learning scenarios (with further experiments presented in App.

Original: G in the supplementary material).
Revised: G in the supplementary material).

Original: First, we consider a ridge regression problem on the CIFAR-10 dataset (Krizhevsky, 2009).
Revised: First, we consider a ridge regression problem on the CIFAR-10 dataset (Krizhevsky, 2009).

Original: Then, we study a case of online convex optimization where the difference between successive minimizers diminishes as time progresses.
Revised: Then, we study a case of online convex optimization where the difference between successive minimizers diminishes as time progresses.

Original: We compare OMMD with the following alternatives: Online Gradient Descent (OGD) (Zinkevich, 2003), Online Multiple Gradient Descent (OMGD) (Zhang et al., 2017), and
Revised: We compare OMMD with the following alternatives: Online Gradient Descent (OGD) (Zinkevich, 2003), Online Multiple Gradient Descent (OMGD) (Zhang et al., 2017), and

Original: Dynamic Mirror Descent (DMD) (Hall & Willett, 2015).
Revised: Dynamic Mirror Descent (DMD) (Hall & Willett, 2015).

Original: In the ﬁrst experiment, we consider multi-class classiﬁcation with ridge regression.
Revised: In the ﬁrst experiment, we consider multi-class classiﬁcation with ridge regression.

Original: In this task, the learner observes a sequence of labeled examples ( ω, z ) , where ω ∈ R d , and the label z , denoting the class of the data example, is drawn from a discrete space Z
Revised: In this task, the learner observes a sequence of labeled examples ( ω, z ) , where ω ∈ R d , and the label z , denoting the class of the data example, is drawn from a discrete space Z

Original: = { 1 , 2 , . .
Revised: = { 1 , 2 , . .

Original: , c } .
Revised: , c } .

Original: We use the CIFARimage dataset, which contains 5 × 10 4 data samples.
Revised: We use the CIFARimage dataset, which contains 5 × 10 4 data samples.

Original: Each data sample ω is a color image of size 32 × 32 pixel that can be represented by a 3072 -dimensional vector, i.e., d = 3072 .
Revised: Each data sample ω is a color image of size 32 × 32 pixel that can be represented by a 3072 -dimensional vector, i.e., d = 3072 .

Original: Data samples correspond to color images of objects, including airplanes, cars, birds, cats, deer, dogs, frogs, horses,[Figure] [Figure] ships, and trucks.
Revised: Data samples correspond to color images of objects, including airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks.

Original: Hence, there are c = 10 different classes.
Revised: Hence, there are c = 10 different classes.

Original: For ridge regression, the cost function associated with each  data point is given by
Revised: For ridge regression, the cost function associated with each batch of data point is given by

Original: [Equation]
Revised: [Equation]

Original: where x is the optimization variable, which is constrained by the set X 
Revised: where x is the optimization variable, which is constrained by the set X = { x : x ∈ R d + ,

Original: = { x : x ∈ R d + , (cid:107) x (cid:107) 1 = 1 }  .
Revised: [Equation] x [Equation] 1 = 1 } , and ( ω t , z t ) represent the batch of data examples at round t .

Original: The goal of the learner is to classify streaming images online by tracking the unknown optimal parameter x ∗ t .
Revised: The goal of the learner is to classify streaming images online by tracking the unknown optimal parameter x ∗ t .

Original: We use the negative entropy regularization function, i.e., r ( x ) = (cid:80) dj
Revised: We use the negative entropy regularization function, i.e., r ( x ) = (cid:80) dj

Original: =1 x j log( x j ) , which is strongly convex with respect to the l 1 -norm.
Revised: =1 x j log( x j ) , which is strongly convex with respect to the l 1 -norm.

Original: Then, the mirror descent update in equationleads to the following update:
Revised: Then, the mirror descent update in equation 4 leads to the following update:

Original: [Equation]
Revised: [Equation]

Original: where x t,j and ∇ f t ( x t ) j denote the j -th component of x t and ∇ f t ( x t ) respectively.
Revised: where x t,j and ∇ f t ( x t ) j denote the j -th component of x t and ∇ f t ( x t ) respectively.

Original: The proof of the above closed-form update is given in App.
Revised: The proof of the above closed-form update is given in App.

Original: F in the supplementary material.
Revised: F in the supplementary material.

Original: In our experiment, we set  α = 0 . 
Revised: In our experiment, we set batch size to 20 data samples per online round, and set α = 0 . 1 .

Original: In Fig.
Revised: In Fig.

Original: 1, we compare the performance of OMMD with DMD, OGD, and OMGD in terms of the dynamic regret.
Revised: 1, we compare the performance of OMMD with DMD, OGD, and OMGD in terms of the dynamic regret.

Original: We see that the methods based on mirror descent perform better than those based on gradient descent as generally expected.
Revised: We see that the methods based on mirror descent perform better than those based on gradient descent as generally expected.

Original: Furthermore, OMMD with M = 10 can reduce the dynamic regret up to 30% in comparison with DMD.
Revised: Furthermore, OMMD with M = 10 can reduce the dynamic regret up to 30% in comparison with DMD.

Original: The dynamic regret associated with all algorithms grow linearly with the number of rounds.
Revised: The dynamic regret associated with all algorithms grow linearly with the number of rounds.

Original: This is because the sequence of minimizers x ∗ t depend on batches of samples that are independent over time, so that they do not converge.
Revised: This is because the sequence of minimizers x ∗ t depend on batches of samples that are independent over time, so that they do not converge.

Original: We note that this is common for online optimization in dynamic settings where steady ﬂuctuation in the environment results in linear dynamic regret.
Revised: We note that this is common for online optimization in dynamic settings where steady ﬂuctuation in the environment results in linear dynamic regret.

Original: Next, we study the performance of OMMD in solving a sequence of quadratic programming problems of the form f t ( x 1 , x 2 ) =
Revised: Next, we study the performance of OMMD in solving a sequence of quadratic programming problems of the form f t ( x 1 , x 2 ) =

Original: ρ (cid:107) x 1 − a t (cid:107) 

Revised: ρ (cid:107) x 1 − a t (cid:107)

Original: + (cid:107) x 2 − b t (cid:107) 2 , where ρ is a positive constant, a t and b t are time-variant vectors, and the decision variable are x 1 ∈ R d 1 , x 2 ∈ R d 2 , such that d 1 + d 2 = d .
Revised: + (cid:107) x 2 − b t (cid:107) 2 , where ρ is a positive constant, a t and b t are time-variant vectors, and the decision variable are x 1 ∈ R d 1 , x 2 ∈ R d 2 , such that d 1 + d 2 = d .

Original: In our experiment, we set ρ = 10 , d 1 = 500 , and d = 1000 .
Revised: In our experiment, we set ρ = 10 , d 1 = 500 , and d = 1000 .

Original: We assume that b t is time-invariant and for all rounds t we have b t = 2 , while a t satisﬁes the recursive formula a t +1 =
Revised: We assume that b t is time-invariant and for all rounds t we have b t = 2 , while a t satisﬁes the recursive formula a t +1 =

Original: a t + 1 / √ t with initial value a 1 = − 1 .
Revised: a t + 1 / √ t with initial value a 1 = − 1 .

Original: We further set the step size α = 0 .
Revised: We further set the step size α = 0 .

Original: We use the same regularization function and constraint set as in the previous experiment.
Revised: We use the same regularization function and constraint set as in the previous experiment.

Original: From Fig.
Revised: From Fig.

Original: 2, we observe that the performance advantage of OMMD is even more pronounced.
Revised: 2, we observe that the performance advantage of OMMD is even more pronounced.

Original: As time progresses and the difference between the successive cost functions becomes less signiﬁcant, the difference between the minimizers decreases.
Revised: As time progresses and the difference between the successive cost functions becomes less signiﬁcant, the difference between the minimizers decreases.

Original: In this case, OMMD can signiﬁcantly improve the performance of online optimization by reducing the gap between the learner’s decisions and the minimizers sequence.
Revised: In this case, OMMD can signiﬁcantly improve the performance of online optimization by reducing the gap between the learner’s decisions and the minimizers sequence.

Original: In particular, compared with DMD, OMMD with M = 10 reduces the dynamic regret up to 80% after 2500 rounds.
Revised: In particular, compared with DMD, OMMD with M = 10 reduces the dynamic regret up to 80% after 2500 rounds.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: The problem of online convex optimization has been extensively studied in the literature since the seminal work of (Zinkevich, 2003).
Revised: The problem of online convex optimization has been extensively studied in the literature since the seminal work of (Zinkevich, 2003).

Original: Most prior works study various online algorithms that guarantee a sublinear bound on the static regret (Zinkevich, 2003; Cesa-Bianchi & Lugosi, 2006; Hazan et al., 2006; Duchi et al., 2010; Shalev-Shwartz, 2012).
Revised: Most prior works study various online algorithms that guarantee a sublinear bound on the static regret (Zinkevich, 2003; Cesa-Bianchi & Lugosi, 2006; Hazan et al., 2006; Duchi et al., 2010; Shalev-Shwartz, 2012).

Original: Here we review the most relevant works with a focus on the dynamic regret.
Revised: Here we review the most relevant works with a focus on the dynamic regret.

Original: 5. D YNAMIC R EGRET OF O NLINE G RADIENT D ESCENT
Revised: 5. D YNAMIC R EGRET OF O NLINE G RADIENT D ESCENT

Original: Dynamic regret was ﬁrst introduced in (Zinkevich, 2003) for the analysis of online gradient descent, where an O ( √ TC T ) bound on the dynamic regret was derived for convex functions.
Revised: Dynamic regret was ﬁrst introduced in (Zinkevich, 2003) for the analysis of online gradient descent, where an O ( √ TC T ) bound on the dynamic regret was derived for convex functions.

Original: When the learner has knowledge of the path length beforehand, the dynamic regret can be upper bounded by O ( √ TC T )
Revised: When the learner has knowledge of the path length beforehand, the dynamic regret can be upper bounded by O ( √ TC T )

Original: (Yang et al., 2016).
Revised: (Yang et al., 2016).

Original: For strongly convex cost functions, the upper bound on the dynamic regret can be  reduced to O ( C T ) (Mokhtari et al., 2016).
Revised: For strongly convex cost functions, the upper bound on the dynamic regret can be [Figure] [Figure] reduced to O ( C T ) (Mokhtari et al., 2016).

Original: The above works make only a single query to the gradient of the cost functions in every round.
Revised: The above works make only a single query to the gradient of the cost functions in every round.

Original: By allowing the learner to make multiple gradient queries per online round, the regret bound can be improved to O (min( C T , S T )) when the cost functions are smooth and strongly convex (Zhang et al., 2017).
Revised: By allowing the learner to make multiple gradient queries per online round, the regret bound can be improved to O (min( C T , S T )) when the cost functions are smooth and strongly convex (Zhang et al., 2017).

Original: The analysis in all aforementioned studies requires the cost functions to be Lipschitz continuous.
Revised: The analysis in all aforementioned studies requires the cost functions to be Lipschitz continuous.

Original: However, many commonly used cost functions do not satisfy this condition over an unbounded feasible set, e.g., the quadratic function, and even when the feasible set is bounded the Lipschitz factor can be excessively large, especially when the cost functions are strongly convex.
Revised: However, many commonly used cost functions do not satisfy this condition over an unbounded feasible set, e.g., the quadratic function, and even when the feasible set is bounded the Lipschitz factor can be excessively large, especially when the cost functions are strongly convex.

Original: Therefore, Lipschitz continuity of the cost functions is not assumed in our analysis.
Revised: Therefore, Lipschitz continuity of the cost functions is not assumed in our analysis.

Original: Instead, we move this condition from the cost functions to the Bregman divergence, which the learner can control and design.
Revised: Instead, we move this condition from the cost functions to the Bregman divergence, which the learner can control and design.

Original: In addition, the above works rely on measuring distances using Euclidean norms, while the updates with Euclidean distance are challenging for some constraint sets, e.g., probability simplex (Duchi, 2018).
Revised: In addition, the above works rely on measuring distances using Euclidean norms, while the updates with Euclidean distance are challenging for some constraint sets, e.g., probability simplex (Duchi, 2018).

Original: It is known that gradient descent does not perform as well as mirror descent, especially when the input dimension is high (Nemirovsky & Yudin, 1983;
Revised: It is known that gradient descent does not perform as well as mirror descent, especially when the input dimension is high (Nemirovsky & Yudin, 1983;

Original: Beck & Teboulle, 2003).
Revised: Beck & Teboulle, 2003).

Original: 5. D YNAMIC R EGRET OF O NLINE M IRROR D ESCENT
Revised: 5. D YNAMIC R EGRET OF O NLINE M IRROR D ESCENT

Original: The dynamic regret of online single-step mirror descent was studied in (Hall & Willett, 2015), where an upper bound of O ( √ T (1 + C T )) was derived for convex cost functions.
Revised: The dynamic regret of online single-step mirror descent was studied in (Hall & Willett, 2015), where an upper bound of O ( √ T (1 + C T )) was derived for convex cost functions.

Original: To take advantage of smoothness in cost functions, an adaptive algorithm based on optimistic mirror descent (Rakhlin &
Revised: To take advantage of smoothness in cost functions, an adaptive algorithm based on optimistic mirror descent (Rakhlin &

Original: Sridharan, 2013) was proposed in (Jadbabaie et al., 2015), which contains two steps of mirror descent per online round.
Revised: Sridharan, 2013) was proposed in (Jadbabaie et al., 2015), which contains two steps of mirror descent per online round.

Original: However, different from our work, in that variant the learner is allowed to make only a single query about the gradient.
Revised: However, different from our work, in that variant the learner is allowed to make only a single query about the gradient.

Original: The algorithm further requires some prior prediction of the gradient in each round, which is used in the second mirror descent step.
Revised: The algorithm further requires some prior prediction of the gradient in each round, which is used in the second mirror descent step.

Original: The dynamic regret bound was given in terms of a combination of the path length C T , deviation between the predictions and the actual gradients D T , and functional variation F T = (cid:80)
Revised: The dynamic regret bound was given in terms of a combination of the path length C T , deviation between the predictions and the actual gradients D T , and functional variation F T = (cid:80)

Original: Tt =1 max x ∈X | f t ( x ) − f t − 1 ( x ) | .
Revised: Tt =1 max x ∈X | f t ( x ) − f t − 1 ( x ) | .

Original: Unfortunately, to achieve this bound, the algorithm requires the design of a time-varying step size that depends on the optimal solution in the previous step, which prevents direct numerical comparison with OMMD.
Revised: Unfortunately, to achieve this bound, the algorithm requires the design of a time-varying step size that depends on the optimal solution in the previous step, which prevents direct numerical comparison with OMMD.

Original: Therefore, in Section 4 we have experimented only with the method of (Hall & Willett, 2015).
Revised: Therefore, in Section 4 we have experimented only with the method of (Hall & Willett, 2015).

Original: All aforementioned works make only a single query to the gradient of the cost functions in every online round.
Revised: All aforementioned works make only a single query to the gradient of the cost functions in every online round.

Original: In contrast, in this work, we allow the learner to make multiple gradient queries per round.
Revised: In contrast, in this work, we allow the learner to make multiple gradient queries per round.

Original: The learner then uses this information to update its decision via multiple steps of mirror descent.
Revised: The learner then uses this information to update its decision via multiple steps of mirror descent.

Original: In this way, we show the dynamic regret can be upper bounded linearly by the minimum among the path length, squared path length, and sum squared gradients.
Revised: In this way, we show the dynamic regret can be upper bounded linearly by the minimum among the path length, squared path length, and sum squared gradients.

Original: Furthermore, as opposed to the aforementioned works, our analysis does not require the cost functions to be Lipschitz continuous.
Revised: Furthermore, as opposed to the aforementioned works, our analysis does not require the cost functions to be Lipschitz continuous.

Original: Finally, there is also recent work in the literature on distributed online mirror descent (Shahrampour &
Revised: Finally, there is also recent work in the literature on distributed online mirror descent (Shahrampour &

Original: Jadbabaie, 2017).
Revised: Jadbabaie, 2017).

Original: As expected, it is more challenging to achieve performance guarantee in distributed optimization.
Revised: As expected, it is more challenging to achieve performance guarantee in distributed optimization.

Original: We focus on centralized online convex optimization in this work.
Revised: We focus on centralized online convex optimization in this work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have studied online convex optimization in dynamic settings.
Revised: We have studied online convex optimization in dynamic settings.

Original: By applying the mirror descent step multiple times in each round, we show that the upper bound on the dynamic regret can be reduced signiﬁcantly from O ( √ T (1 + C T )) to O (min( C T , S T , G T )) , when the cost functions are strongly convex and smooth.
Revised: By applying the mirror descent step multiple times in each round, we show that the upper bound on the dynamic regret can be reduced signiﬁcantly from O ( √ T (1 + C T )) to O (min( C T , S T , G T )) , when the cost functions are strongly convex and smooth.

Original: In contrast to prior studies (Hall & Willett, 2015; Zhang et al., 2017;
Revised: In contrast to prior studies (Hall & Willett, 2015; Zhang et al., 2017;

Original: Mokhtari et al., 2016), our analysis does not require the cost functions to be Lipschitz continuous.
Revised: Mokhtari et al., 2016), our analysis does not require the cost functions to be Lipschitz continuous.

Original: Numerical experiments with the CIFAR-10 dataset, and sequential quadratic programming, and additional examples show substantial improvement on the dynamic regret compared with existing alternatives. 
Revised: Numerical experiments with the CIFAR-10 dataset, and sequential quadratic programming, and additional examples show substantial improvement on the dynamic regret compared with existing alternatives. A A DDITIONAL D EFINITIONS

Revised: Deﬁnition 3 : A function f ( · ) is L -smooth, if there exists a positive constant L such that

Revised: [Equation]

Revised: Deﬁnition 4 : A convex function f ( · ) is λ -strongly convex with respect to some norm (cid:107) · (cid:107) , if there exists a positive constant λ such that

Revised: [Equation]

Revised: Deﬁnition 5 : A function f ( · ) is Lipschitz continuous with factor G if for all x and y in X , the following holds:

Revised: [Equation]

Revised: B P ROOF OF L EMMA

Revised: Consider single-step mirror descent update as follows:

Revised: [Equation]

Revised: Strong convexity of the above minimization objective implies

Revised: [Equation]

Revised: Furthermore, from the smoothness condition, we have

Revised: [Equation]

Revised: Substituting equation 9 into equation 8, and setting y = x ∗ t , we obtain

Revised: [Equation]

Revised: Since α ≤ 1 L

Revised: [Equation]

Revised: and regularization function r ( · ) is 1 -strongly convex

Revised: [Equation]

Revised: we have

Revised: [Equation]

Revised: [Equation]

Revised: Next, we exploit the strong convexity of the cost function, i

Revised: [Equation]

Revised: e.,

Revised: [Equation]

Revised: Combining equation 10, equation

Revised: [Equation]

Revised: and equation

Revised: [Equation]

Revised: we obtain

Revised: [Equation]

Revised: Next, we use the result of (Hazan & Kale, 2014), which states that for evey λ -strongly convex function f t ( . ) , the following bound holds:

Revised: [Equation]

Revised: where x ∗ t = argmin x ∈X f t ( x ) .

Revised: Combining the above with equation 13, we obtain

Revised: [Equation]

Revised: where β

Revised: = 1 − 2 λα 1+ λα .

Revised: (cid:3)

Revised: C P ROOF OF T HEOREMC. K EY L EMMAS

Revised: The following two lemmas pave the way for our regret analysis leading to Theorem 2.

Revised: Lemmapresents an alternative form for the mirror descent update.

Revised: Lemma 7 Suppose there exists z t +1 that satisﬁes ∇ r ( z t +1 )

Revised: = ∇ r ( x t ) − α ∇ f t ( x t ) , for some strongly convex function r ( · ) , and step size α .

Revised: Then, the following updates are equivalent

Revised: [Equation]

Revised: Proof .

Revised: We begin by expanding equation 16 as follows:

Revised: [Equation]

Revised: Thus, the update in equation 16 is equivalent to equation 17.

Revised: [Equation]

Revised: Lemma 8 Under the same convexity and smoothness condition stated in Theorem 2, let x t be the sequence of decisions generated by OMMD.

Revised: Then, the following bound holds:

Revised: [Equation]

Revised: where L r is the smoothness factor of the regularization function r ( · ) , and β is the shrinking factor obtained in Lemma 1.

Revised: Proof .

Revised: Using the result of Lemma 1, OMMD with M mirror descent steps guarantees

Revised: [Equation]

Revised: Since the regularization function r ( · ) is 1 -strongly convex, we have

Revised: [Equation]

Revised: Next, we exploit the smoothness condition of the regularization function r ( · ) , i.e.,

Revised: [Equation]

Revised: [Equation]

Revised: By combining the above with equation 20, and equation 21, and using the deﬁnition of Bregman divergence, we obtain

Revised: [Equation]

Revised: Taking the square root on both sides of equation 23 completes the proof.

Revised: [Equation]

Revised: C. P ROOF OF THE T HEOREM

Revised: Now, we are ready to present the proof of Theorem 2.

Revised: In this proof, we will use the following properties of Bregman divergence.

Revised: (a)

Revised: By direct substitution, the following equality holds

Revised: for

Revised: any

Revised: x, y

Revised: z ∈

Revised: X ,

Revised: [Equation]

Revised: (b

Revised: If x = argmin x

Revised: [Equation]

Revised: ∈X D r ( x

Revised: [Equation]

Revised: , z ) , i.e., x is the Bregman projection of z into the

Revised: set

Revised: X ,

Revised: then

Revised: for any arbitrary point y

Revised: ∈

Revised: X ,

Revised: we

Revised: have

Revised: [Equation]

Revised: To bound the dynamic regret

Revised: [Equation]

Revised: we begin by using the strong convexity of the cost function f t ( · ) ,

Revised: i

Revised: [Equation]

Revised: e., f t

Revised: [Equation]

Revised: where in the second line we have used the alternative mirror descent update stated in Lemma 7, i.e., ∇ f t ( x t ) =

Revised: (1 /α )( ∇ r ( x t ) − ∇ r ( z t +1 )) .

Revised: To obtain the third line, we have utilized the Bregman divergence property in equation 24.

Revised: We have used the Bregman projection property in equation 25 in the fourth line.

Revised: By omitting some negative terms, and using equation 14, we obtain the right-hand side of equation 26.

Revised: Thus, if α > 12 λ , we have

Revised: [Equation]

Revised: (27) where we have used the Lipschitz continuity of Bregman divergence to obtain inequality (a), and we have applied Lemma 8 to obtain inequality (b).

Revised: Summing equation 27 over time, we have

Revised: [Equation]

Revised: Now, we proceed to bound (cid:80) Tt =1 (cid:107) x

Revised: t − x ∗ t (cid:107) as follows:

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: where we used the result of Lemma 8 to obtain inequality (a).

Revised: If M ≥ (cid:100) (cid:0)

Revised: [Equation]

Revised: [Equation]

Revised: log L r

Revised: [Equation]

Revised: , we have

Revised: [Equation]

Revised: which implies L r β M < 1 .

Revised: Therefore, by combining equation 29 and equation 30, we have

Revised: [Equation]

Revised: Finally, substituting equation 31 into equation 28 completes the proof.

Revised: [Equation]

Revised: D P ROOF OF T HEOREM

Revised: In order to bound the dynamic regret

Revised: [Equation]

Revised: we begin by the smoothness condition of the cost function f t ( . )

Revised: , i

Revised: [Equation]

Revised: e.,

Revised: [Equation]

Revised: Next, we use the fact

Revised: [Equation]

Revised: for any arbitrary positive constant θ > 0 .

Revised: Thus

Revised: [Equation]

Revised: we have

Revised: [Equation]

Revised: Summing equation 34 over time, we obtain

Revised: [Equation]

Revised: Now

Revised: [Equation]

Revised: we proceed by bounding (cid:80)

Revised: Tt =1 (cid:107) x

Revised: t − x ∗ t (cid:107) 2 as follows:

Revised: [Equation]

Revised: We note that if M ≥ (cid:100) (cid:0) 1 2 + 1 2 αλ (cid:1) log 2 L r (cid:101) , then 2 β M L r < 1 .

Revised: Therefore, from equation 36 we can obtain

Revised: [Equation]

Revised: Substituting equation 37 into equation 35 completes the proof.

Revised: [Equation]

Revised: E P ROOF OF T HEOREM

Revised: The proof of Theorem 5 initially follows the ﬁrst half of the proof of Theorem 2, which is repeated here for completeness.

Revised: To analyze the dynamic regret

Revised: [Equation]

Revised: we ﬁrst use the strong convexity of the cost function f t ( · ) , i.e.,

Revised: [Equation]

Revised: [Equation]

Revised: where in the second line we have used the alternative mirror descent update stated in Lemma 7, i.e.,

Revised: [Equation]

Revised: /α )( ∇ r ( x t ) − ∇ r ( z t +1 )) .

Revised: To obtain the third line, we have utilized the Bregman divergence property in equation 24.

Revised: We have used the Bregman projection property in equation 25 in the fourth line.

Revised: By omitting some negative terms, and using equation 14, we obtain the right-hand side of equation 38.

Revised: Therefore

Revised: [Equation]

Revised: if α > 12 λ , we have

Revised: [Equation]

Revised: Now we continue to bound D r ( x t , z t +1 ) .

Revised: By the deﬁnition of Bregman divergence, we have

Revised: [Equation]

Revised: The strong convexity of the regularization function implies

Revised: [Equation]

Revised: [Equation]

Revised: Combining the above with equation

Revised: [Equation]

Revised: we obtain

Revised: [Equation]

Revised: By substituting equation 42 into equation 40, and summing over time, we have

Revised: [Equation]

Revised: F C LOSED - FORM U PDATE FOR M IRROR D ESCENT

Revised: In this section, we derive the close-form mirror descent update in equation 6.

Revised: Let r ( y ) =

Revised: (cid:80) dj

Original: {+=
+}
Revised: {+=
+}

Revised: y j log( y j ) be the negative entropy.

Revised: Then, we have

Revised: [Equation]

Revised: where y it,j denotes the j -th component of the decision vector y it , and D KL ( y, y it ) represents the KL divergence between y and y it .

Revised: Now consider the update in equation 5, which can be written as follows:

Revised: [Equation]

Revised: subject to (cid:104) 1 , y (cid:105)

Revised: = 1 , y ≥ 0 .

Revised: The Lagrangian of the above problem is given by

Revised: [Equation]

Revised: where λ ∈ R and γ ∈ R d + are Lagrange multipliers corresponding to the constraints.

Revised: Next, we take derivative with respect to y to obtain

Revised: [Equation]

Revised: [Equation]

Revised: Setting the above to zero results in the following closed-form update:

Revised: [Equation]

Revised: G A DDITIONAL E XPERIMENTS

Revised: In this section, we present additional experiments to study the performance of OMMD.

Revised: In the ﬁrst experiment, we use the MNIST dataset.

Revised: In the second experiment, we consider a switching problem where the cost function switches between two quadratic functions after a speciﬁc number of rounds.

Revised: First, we consider the well-known MNIST digits dataset, where every data sample ω is an image of size 28 × 28 pixel that can be represented by a 784 -dimensional vector, i.e., d = 784 .

Revised: Each sample corresponds to one of the digits in { 0 , 1 , . .

Revised: , 9 } , and thus, there are c = 10 different classes.

Revised: The goal of the learner is to classify streaming digit images in an online fashion.

Revised: We consider a robust regression problem, where the cost function for the batch of data samples at time t is given by

Revised: [Equation]

Revised: where x is the optimization variable, belonging to the constraint set is X

Revised: = { x : x ∈ R n + , (cid:107) x (cid:107) 1 = 1 } .

Revised: We use the negative entropy regularization function, i.e., r ( x ) = (cid:80) di

Revised: =1 x i log( x i ) , which is strongly convex with respect to the l 1 − norm.

Revised: We set the step size α = 0 .

Revised: 1 and use a batch size of 20 data examples per round.

Revised: From Fig.

Revised: 3, we again observe that OMMD consistently outperforms the other alternatives.

Revised: In particular, compared with DMD, applying M = 10 steps of mirror descent can reduce the dynamic regret up to 20% .

Revised: We also see that the dynamic regret grows linearly with the number of rounds,

Revised: [Figure]

Revised: [Figure]

Revised: which is a natural consequence of steady ﬂuctuation in the sequence of dynamic minimizers x ∗ t as explained before.

Revised: Next, we consider the case where the cost function switches between two functions.

Revised: Both functions are in the quadratic form f t ( x ) = (cid:107)

Revised: A t x − b t (cid:107) 22 , where A t ∈ R d × d , and b t ∈ R d .

Revised: In particular, we assume that the parameter A t is chosen among

Revised: A (1) t

Revised: [Equation]

Revised: (cid:124)

Revised: [Equation]

Revised: (cid:125)

Revised: d

Revised: [Equation]

Revised: ,(cid:124) (cid:123)(cid:122) (cid:125) d) , and

Revised: A (2) t

Revised: [Equation]

Revised: d) , such that d 1 + d 2 = d , and b t =

Revised: [ 1 t p 2 , . .

Revised: , 1 t p 2 ] (cid:48) .

Revised: Therefore, at each round the cost function is either f (1) t ( x )

Revised: = (cid:107) A (1) t x − b t (cid:107) 22 or f (2) t ( x ) = (cid:107) A (2) t x − b t (cid:107) 22 .

Revised: We assume that the cost function switches between f (1) t ( · ) and f (2) t ( · ) every τ rounds.

Revised: In our experiment, we set d 1 = 10 , d = 1000 , p 1 = 0 . 9 , and p 2 = 0 .

Revised: We further set the switching period τ = 10 , and parameter α = 0 .

Revised: The dynamic regret roughly reﬂects the accumulated mismatch error over time.

Revised: In Fig.

Revised: 4, we compare the performance of OMMD with that of other alternatives in terms of the dynamic regret.

Revised: OMMD with M = 10 nearly halves the dynamic regret of DMD after 300 rounds.

Revised: Furthermore, the beneﬁt of applying multiple steps of mirror descent can be signiﬁcant even for smaller values of M .