M ULTI
-O BJECTIVE
O
NLINE L
EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper presents a systematic study of multi-objective online learning.
We first formulate the framework of Multi-Objective Online Convex Optimization, which encompasses a novel multi-objective dynamic regret in the unconstrained max-min form.
We show that it is equivalent to the regret commonly used in the zero-order multi-objective bandit setting and overcomes the problem that the latter is hard to optimize via first-order gradient-based methods.
Then we propose the Online Mirror Multiple Descent algorithm with two variants, which computes the composite gradient using either the vanilla min-norm solver or a newly designed L 1 -regularized min-norm solver.
We further derive regret bounds of both variants and show that the L 1 -regularized variant enjoys a lower bound.
Extensive experiments demonstrate the effectiveness of the proposed algorithm and verify the theoretical advantage of the L 1 -regularized variant
I NTRODUCTION
Traditional optimization methods for machine learning are usually designed to optimize a single objective.
However, in many real-world machine learning applications, we are often required to optimize multiple correlated objectives simultaneously.
For example, in autonomous driving (Huang et al., 2019; Lu et al., 2019b), the self-driving vehicle needs to learn to solve multiple tasks such as self-localization and object identification at the same time.
In online advertising (Ma et al., 2018a;b), the advertiser aims to choose the exposure of items to different users so that both the Click-Through
Rate (CTR) and the Post-Click Conversion Rate (CVR) are maximized simultaneously.
In many multi-objective scenarios, the objectives may conflict with each other (Kendall et al., 2018).
Hence, there may not exist any single solution that optimizes all the objectives simultaneously.
For example, in the online advertising scenario, merely optimizing either CVR or CTR will incur the degradation of performance of the other (Ma et al., 2018a;b).
Multi-objective optimization (MOO) (Marler and Arora, 2004; Deb, 2014) is concerned with optimizing multiple conflicting objectives simultaneously.
Many different approaches for MOO have been proposed, which include evolutionary methods (Murata et al., 1995; Zitzler and Thiele, 1999), scalarization methods (Fliege and Svaiter, 2000) and gradient-based iterative methods (D ´ esid ´ eri, 2012).
Recently, due to the great success of training multi-task deep neural networks, the first-order gradient-based iterative methods, i.e., Multiple Gradient Descent Algorithm (MGDA) and its variants, have regained a significant amount of research interest (Sener and Koltun, 2018; Lin et al., 2019; Yu et al., 2020).
These methods compute a composite gradient based on the gradient information of all the individual objectives and then apply the composite gradient to model update.
The determination of the composite gradient is based on a min-norm solver (D´esid´eri, 2012) which yields the common descent direction of all the objectives.
However, compared to the increasingly wide application prospect, the first-order gradient-based iterative algorithms are relatively understudied, especially for the online learning setting.
Multiobjective online learning is very important due to reasons in two main aspects.
First, due to the data explosion in many real-world scenarios such as various web applications, making in-time prediction requires to perform online learning.
Second, the theoretical investigation will lay a solid foundation for the design of new optimizers of multi-task deep neural networks, such as multi-objective Adam.
In this paper, we conduct a systematic study of multi-objective online learning.
To begin with, we first formulate the framework of Multi-Objective Online Convex Optimization (MO-OCO).
The biggest challenge in the design of MO-OCO is the derivation of an appropriate regret definition in the multi-objective setting.
Since the multiple objectives form a vector space, we need a discrepancy metric to scalarize the loss vector.
Specifically, we adopt the Pareto suboptimality gap (PSG), which is a distance-based discrepancy metric extensively used in multi-objective bandits (Turgay et al., 2018; Lu et al., 2019a).
Then analogously to the single-objective online setting, we define the multi-objective static regret and the multi-objective dynamic regret.
However, since PSG is a metric motivated purely from the geometric view, it is intrinsically difficult to directly optimize using firstorder gradient-based iterative methods.
To remedy this problem, for the multi-objective dynamic regret, via a highly non-trivial transformation, we derive its equivalent regret in the unconstrained max-min form, which is much easier to optimize.
Unfortunately, we further show that such an equivalence does not hold for the multi-objective static regret since PSG always yields non-negative measurements.
Hence, in this paper, we mainly focus on studying the multi-objective dynamic regret and leave the complete treatment of its static counterpart as an open problem.
Based on the proposed MO-OCO framework, we develop the Online Mirror Multiple Descent (OMMD) algorithm.
The key module of OMMD is the gradient composition scheme, which utilizes the information of all the individual gradients to compute a composite gradient that tends to descend all the losses simultaneously.
By directly applying the min-norm solver (D´esid´eri, 2012) in the offline setting to determine the composition weights, we give the first variant of OMMD termed as OMMD-I.
However, the min-norm solver only uses the instantaneous gradients and ignores the historical information, which can be very unstable in the online setting where the losses and the gradients at different rounds can vary wildly.
To make the learning process more stable, we introduce a carefully designed L 1 -regularizer to the vanilla min-norm solver, which results in the second variant of OMMD, namely OMMD-II.
We then give the theoretical analysis of the proposed OMMD algorithm.
Specifically, we derive a non-trivial dynamic regret bound of OMMD-II, which includes that of OMMD-I as a special case.
The dynamic regret bound is in the same order
O
( V 1 / 3 T
T 2 / 3 ) , where T is the time horizon and V T is the temporal variability at T , as that of its single-objective counterpart(Zhang et al., 2018).
We further show that the regret bound of OMMD-II is lower than that of OMMD-I.
To evaluate the effectiveness of our proposed algorithm, we conduct extensive experiments using both simulation datasets and real-world datasets.
Specifically, we first design simulation experiments to verify the capability of OMMD-II to track the dynamic online data streams.
Then we successfully realize adaptive regularization for online learning using our MO-OCO formalism, which demonstrates the effectiveness of OMMD-II in the convex setting.
We further conduct online multitask learning experiments with deep neural networks, the results of which show that both OMMD-I and OMMD-II are effective in the non-convex setting.
Moreover, in both simulation and multi-task deep experiments, OMMD-II yields better performance than OMMD-I, which verifies the theoretical superiority of the regularized min-norm method over the vanilla min-norm method.
In summary, in this paper, we give the first systematic study of multi-objective online learning, which includes novel framework, algorithm design and theoretical analysis.
We believe that our paper paves the way for future research on multiple-objective optimization and multi-task learning.
P RELIMINARIES
In this section, we briefly review the necessary background knowledge of online convex optimization and multi-objective optimization.
2. O NLINE C ONVEX O PTIMIZATION
Online Convex Optimization (OCO) (Zinkevich, 2003; Hazan, 2019) is the most commonly adopted framework for designing online learning algorithms.
It can be viewed as a structured repeated game between a learner and an adversary.
At each round t ∈ { 1 , . . .
, T } , the learner is required to generate a decision x t from a convex compact set
X ⊂ R n .
Then the adversary replies the learner with a convex function f t : X → R and the learner suffers the loss f t ( x t ) .
The goal of the learner is to minimize the regret with respect to the best fixed decision in hindsight, i.e.,
[Equation]
Note that the above regret is the static regret (Hall and Willett, 2013), which compares the learner’s cumulative loss with that of a fixed decision.
There is another version of regret, namely the dynamic regret (Hall and Willett, 2013; Zhang et al., 2018), which compares the learner’s cumulative loss with that of a sequence of changing decisions, i.e.,
[Equation]
Any meaningful regret is required to be sublinear in T , i.e., lim T →∞ R S/D ( T ) /T
= 0 , which implies that when T is large enough, the learner can perform as well as the best fixed decision in hindsight (for static regret) or the changing optimal decisions at each round (for dynamic regret).
Online Mirror Descent (OMD) (Hazan, 2019) is a classic first-order online learning algorithm.
At each round t ∈ { 1 , . . .
, T } , OMD yields its decision using the following formula
[Equation]
where η is the step size, R : X → R is the regularization function, and B R ( x, x ′ )
=
R ( x ) −
R ( x ′ )
− ⟨∇ R ( x ′ ) , x − x ′ ⟩ is the Bregman divergence induced from R .
As a generic algorithm, by instantiating different regularization functions, OMD can induce two important algorithms, i.e., Online Gradient Descent (Zinkevich, 2003) and Online Exponentiated Gradient (Hazan, 2019).
Several papers work on the dynamic regret of online mirror descent (Jadbabaie et al., 2015; Shahrampour and Jadbabaie, 2017).
2. M ULTI -O BJECTIVE O PTIMIZATION
Multiple-objective optimization (MOO) is concerned with solving the problems of optimizing multiple objective functions simultaneously (Zitzler and Thiele, 1999; Sener and Koltun, 2018).
In general, since different objectives may conflict with each other, there is no single solution that can optimize all the objectives at the same time.
Instead, MOO seeks to find solutions that achieve Pareto optimality.
In the following, we exposit Pareto optimality and related definitions more formally using a vector-valued loss H =
( h 1 , . . .
, h m ) ⊤ as objectives, where m ≥ 2 and h
i : K → R , i ∈ { 1 , . . .
, m } , K ⊂ R , is the i -th loss function.
Definition 1 ( Pareto optimality ) .
(a)
For any two solutions x, x ′ ∈ K , we say that x dominates x ′ , denoted as x ≺ x ′ or x ′ ≻
x , if h i ( x ) ≤ h
i ( x ′ ) for all i , and there exists one i such that h i ( x ) <
h i ( x ′ ) ; otherwise, we say that x does not dominate x ′ , denoted as x ⊀ x ′ or x ′ ⊁ x .
(b) A solution x ∗ ∈ K is called Pareto optimal if it is not dominated by any other solution in K .
There may exist multiple Pareto optimal solutions.
For example, it is easy to show that the optimizer of any single objective, say, x ∗ 1 ∈ arg min x ∈K h 1 ( x ) , is Pareto optimal.
Different Pareto optimal solutions reflect different trade-offs among the objectives (Sener and Koltun, 2018; Lin et al., 2019).
Definition 2 ( Pareto front ) .
(a) All Pareto optimal solutions form the Pareto set, denoted as P K ( H ) .
(b) The image of P K ( H ) constitutes the Pareto front, denoted as P ( H )
= { H ( x ) | x ∈ P K ( H ) } .
Now that we’ve established the notion of optimality in MOO, we proceed to introduce the metrics that measure the discrepancy of an arbitrary solution x ∈ K from being optimal.
Recall that, in the single-objective setting with merely one loss function h : Q → R , where Q ⊂ R , for any z ∈ Q , the loss gap h
( z ) − min z ′′ ∈Q h ( z ′′ ) is directly the discrepancy measure.
However, in MOO with more than one loss, for any x ∈ K , the loss gap H ( x ) − H ( x ′′ ) , where x ′′ ∈ P K ( H ) , is a vector.
Intuitionally, the desired discrepancy metric shall scalarize the vector-valued loss gap and yield the value 0 for any Pareto optimal solution.
In general, there are two commonly used discrepancy metrics in MOO, namely Pareto suboptimality gap (PSG) (Turgay et al., 2018) and Hypervolume (HV) (Bradstreet, 2011).
As HV is a volume-based metric, it is very difficult to optimize or analyze via iterative algorithms (Zhang and Golovin, 2020).
Hence in this paper, we adopt PSG, which has been extensively used in multi-objective bandits (Turgay et al., 2018;
Lu et al., 2019a).
Definition 3 ( Pareto suboptimality gap ) .
For any x ∈ K , the Pareto suboptimality gap to a given comparator set K ∗ ⊂ K , denoted as ∆( x ; K ∗ , H ) , is defined as the minimal scalar ϵ ≥ 0 that needs to be subtracted from all entries of H ( x ) , such that H ( x ) − ϵ 1 is not dominated by any point in K ∗ , where 1 denotes the all-one vector in R m , i.e.,
[Equation]
Clearly, PSG is a distance-based discrepancy metric that motivated from a purely geometric viewpoint.
In practice, the comparator set K ∗ is often set to be the Pareto set P K ( H ) (Turgay et al., 2018).
Then for any x ∈ K , its PSG is always non-negative and equals to zero if and only if x ∈ P K ( H ) .
Multiple Gradient Descent Algorithm (MGDA) is an offline first-order algorithm for MOO (Fliege and Svaiter, 2000; D´esid´eri, 2012).
At each iteration l ∈ { 1 , . . .
, L } , where L is the maximum number of iterations, it first computes the gradient ∇ h
i ( x l ) for each objective i ∈ { 1 , . . . , m } , and then derive the composite gradient g l = (cid:80) mi =1 λ il ∇ h i ( x l ) as the convex combination of these multiple gradients; it applies the composite gradient to execute the gradient descent step to update the decision, i.e., x l +1 = x
l − ηg l , where η is the step size.
The core module of MGDA is the determination of the weights λ l = ( λ 1 l , . . .
, λ ml )
for the gradient composition, which is given as
[Equation]
where ∆ m = { λ ∈ R m | λ i ≥ 0 , i ∈ { 1 , . .
, m } , and (cid:80) mi
=
λ i = 1 } denotes the probabilistic simplex in R m .
This is a min-norm solver which finds the weights in the simplex that yields the minimum L 2 norm of the composite gradient.
Thus MGDA is also called the min-norm method.
Existing works (D ´ esid ´ eri, 2012; Sener and Koltun, 2018) have shown that MGDA is guaranteed to decrease all the objectives simultaneously until it reaches a Pareto optimal decision (under the convex setting where all h i are convex functions).
M ULTI -O BJECTIVE O NLINE C ONVEX O PTIMIZATION
In this section, we formally formulate the framework of multi-objective optimization in the online setting, termed Multi-Objective Online Convex Optimization (MO-OCO).
We tailor the famous online convex optimization (OCO) framework to the multi-objective setting, which can be viewed as a repeated game between an online learner and the adversarial environment.
At each round t ∈ { 1 , . . .
, T } , the learner generates a decision x t from a given convex compact decision set X ⊂ R n .
Then the adversary replies the decision with a vectoral loss function F t ( x ) :
X → R m , where its i -th component f it ( x ) :
X → R belongs to the i -th objective, and the learner suffers the loss F t ( x t ) ∈ R m .
The goal of the learner is to generate a sequence of decisions { x t | 1 ≤ t ≤ T } so that the cumulative loss (cid:80) Tt =1 F t ( x t ) can be optimized.
Recall that, in the single-objective setting, the performance metric R ( T ) =
(cid:80) Tt =1 f t ( x t ) − f t ( x ∗ t ) , i.e., the regret, compares the actual decisions x t with some comparator x ∗ t ∈ X at each round t ∈ { 1 , . .
, T } .
In general, there are two common types of regret which differ in the comparators, i.e., the static regret and the dynamic regret.
For the static regret, all comparators x ∗ t , ∀ t ∈ { 1 , . . .
, T } are identically set as the fixed optimal decision x ∗ w.r.t.
all losses in hindsight, i.e., x ∗ t ≡ x ∗ ∈ arg min x ∈X (cid:80) Tt =1 f t ( x ) .
For the dynamic regret, the comparator x ∗ t at each round t is selected as the optimal decision w.r.t.
the instantaneous loss f t at that round, i.e., x ∗ t ∈ arg min x ∈X f t ( x ) .
In analogy, in the multi-objective setting, we define the regret as R ( T ) =
(cid:80) Tt =1 ∆ t , where the quantity ∆ t at each round t compares the actual decisions x t with some comparator x ∗ t ∈ X .
However, in general, no single decision can optimize all the objectives at the same time.
Hence, it is reasonable to compare x t with all the Pareto optimal decisions that constitute a comparator set
X ∗ t ⊂ X .
Specifically, we introduce the Pareto suboptimality gap (PSG) (Turgay et al., 2018), i.e.,
[Equation]
Then the multi-objective regret can be defined as R ( T ) =
(cid:80) Tt =1 ∆( x t ; X ∗ t , F t ) .
Given the above definitions, we can formulate the multi-objective variants of the static and dynamic regret respectively, by using different comparator set X ∗ t at each step.
Specifically, if we set all X ∗ t to be the
Pareto set of the cumulative loss (cid:80)
Tt =1 F t , then we can formulate a regret metric termed the multiobjective static regret (recall that P X ( F ) denotes the Pareto set of F )
R MOS ( T ) :=
T (cid:88) t = ∆( x t ; X ∗ , F t ) , where X ∗ =
P X (
T (cid:88) t =
F t ) .
Alternatively, if we set X ∗ t to be the Pareto set of the instantaneous loss F t at each round t , then we can give a regret metric termed the multi-objective dynamic regret
[Equation]
where X ∗ t =
P X ( F t ) , ∀ t ∈ { 1 , . . .
, T } .
Recall that, PSG is a zero-order metric motivated in a purely geometric sense, namely, its calculation can be viewed as a constrained minimization problem (1) with an unknown boundary f it ( x ′′ ) , ∀ x ′′ ∈
X ∗ t .
Hence, it is intrinsically complex to design a first-order algorithm to optimize PSG given its unknown constraints, not to mention the regret analysis.
Surprisingly, specific to the multi-objective dynamic regret R MOD , we can transform it into an unconstrained max-min form as follows.
The derivation utilizes Pareto optimality of X ∗ t and is highly non-trivial, which is given in the appendix due to the space limit.
The equivalent form is closely related to the dynamic regret, as it recovers the dynamic regret defined for λ t ⊤ F t , t ∈ { 1 , . . .
, T } if we determine λ t at each round t ∈ { 1 , . . .
, T } beforehand.
Moreover, it exactly reduces to the standard dynamic regret R D in the single-objective setting.
Proposition 1.
The multi-objective dynamic regret has an equivalent form, i.e.,
[Equation]
where ∆ m represents the probabilistic simplex in R m .
Remark.
In the single-objective setting where m = 1 , the probabilistic simplex ∆ m collapses into a single point { 1 } .
Moreover, the Pareto set X ∗ t of the scalar loss function F t : X → R reduces to the optimal set arg min x ∈X F t ( x ) .
Hence we have R MOD ( T ) =
(cid:80) Tt =1 ( F t ( x t )
− min x ∈X F t ( x )) , which is exactly the dynamic regret R D ( T ) in the standard online setting.
Unfortunately, for the multi-objective static regret R MOS , such a correspondence does not exist.
Here is the reason.
In R MOS , the comparator set X ∗ is the Pareto set of the cumulative loss (cid:80)
Tt =1 F t rather than the instantaneous loss F t .
Hence, at some specific round t , the actual decision x t may Pareto dominate all decisions in X ∗ w.r.t.
the instantaneous F t , and so we expect the discrepancy metric ∆ t to give a negative measurement.
However, PSG (as well as other commonly used discrepancy metrics such as Hypervolume) is always non-negative, so the induced R MOD is not aligned with R S .
For example, when m = 1 , we have R MOS ( T ) =
sup x ∗ ∈X ∗ (cid:80)
Tt =1 max { F t ( x t ) − F t ( x ∗ ) , 0 } , which can be much looser than the static regret R S ( T ) =
sup x ∗ ∈X ∗ (cid:80) Tt =1 F t ( x t ) − F t ( x ∗ ) .
Therefore, the analysis of R MOD is intrinsically complex if we only use existing discrepancy metrics; its analysis may require a completely new discrepancy metric ∆ t that allows to give a negative measurement.
Given the limits of existing discrepancy metrics to characterize R MOS , in this paper, we mainly focus on the dynamic variant R MOD .
Indeed, the algorithm design and theoretical analysis w.r.t.
R MOD are already highly non-trivial.
We will leave the analysis of R MOS for future works.
O NLINE M IRROR M ULTIPLE D ESCENT
In this section, we first present the Online Mirror Multiple Descent (OMMD) algorithm, then provide theoretical analysis for it.
4. T HE A LGORITHM
The protocol of OMMD is given in Algorithm 1.
At each round t , the learner computes the gradient for each loss ∇ f i t ( x t ) , then determines the weights for the composition of these multiple gradients, and finally executes an online mirror descent step using the composite gradient.
For simplicity, we define the matrix form of the multiple gradients as ∇ F t ( x t ) =
[ ∇ f 1 t ( x t ) , . .
. , ∇ f mt ( x t )]
∈ R n × m .
The core module of OMMD is the composition of the multiple gradients.
In intuition, the composite gradient g t shall elaborately utilize the information of all the individual gradient.
As illustrated in
Preliminary (see the MGDA part), in the offline setting, there is one simple yet effective scheme, weights for the gradient composition
[Equation] [Equation]
[Equation] [Equation]
i.e., the min-norm method (D ´ esid ´ eri, 2012; Sener and Koltun, 2018), which computes a common descent direction that can descend all the losses simultaneously.
We directly apply it to the online setting, which results in the first variant of OMMD, i.e., OMMD-I, as shown in Algorithm 1.
Despite its simplicity, OMMD-I may not yield optimal decisions in the online setting, since the min-norm method is designed in the offline setting, which does not capture the characteristics of the online setting.
Specifically, the composition weights λ t given by the min-norm solver are determined solely by the gradients at the instantaneous round t , regardless of the historical information.
Indeed, this makes sense in the offline setting, where the optimized loss function does not change too much.
In the online setting, however, the losses at different rounds can vary wildly and so are the gradients consequently.
Hence, directly applying the min-norm method will yield very different λ t s at different rounds, which makes the learning process unstable and even fail to converge.
To fix this issue, we propose to add a regularizer r ( λ, λ t − 1 ) to the min-norm solver when determining the composition weights λ t , where λ t − 1 denotes the composition weights at the precedent round.
Such a regularizer ensures that the new weights λ t will not move too far away from the precedent weights λ t − 1 .
In principle, r ( λ, λ t − 1 ) can take many forms such as L 1 -norm, L 2 -norm and KL divergence etc.
Here we use the L 1 norm since it aligns well with the simplex constraint of λ .
This L 1 -regularized min-norm method results in the second variant of OMMD, i.e., OMMD-II, as shown in Algorithm 1.
Later we will show that OMMD-II attains a lower theoretical regret bound than OMMD-I, and it is also much more robust in experiments.
4. A NALYSIS
We first provide a general bound for OMMD, which is agnostic of the choice of λ t at each round.
Assumption 1 ( Bregman divergence ) .
The regularization function R is 1 -strongly convex with respect to a norm ∥ · ∥ .
In addition, the Bregman divergence is γ -Lipschitz continuous, i.e.,
B R ( x, z ) − B R ( y, z ) ≤ γ ∥ x − y ∥ , ∀ x, y, z ∈ dom R, where dom R denotes the domain of the regularization function R and satisfies X ⊂ dom R ⊂ R n .
Lemma 1.
Suppose the diameter of the decision set X is bounded by D .
Assume F t is bounded, i.e., | f it ( x ) | ≤ F for any x ∈ X , t ∈ { 1 , . .
, T } , i ∈ { 1 , . . .
, m } .
Then for any δ ∈ { 1 , . . .
, T } ,
OMMD-I or OMMD-II with composition weights λ t at each round t attains the following regret R MOD [Equation] T
[Equation]
≤
[Equation] δ [Equation] The bound can be further simplified under the assumptions of temporal variability and Lipschitz continuity, which are commonly used in dynamic regret analysis (Besbes et al., 2015; Yang et al., 2016; Campolongo and Orabona, 2021).
Assumption 2 ( Temporal variability ) .
For each i ∈ { 1 , . . .
, m } , there exists some positive and finite V T such that (cid:80) T − 1 t =1 sup x ∈X | f it ( x ) [Equation] V [Equation]
Assumption 3 ( Lipschitz continuity ) .
For each i ∈ { 1 , . . .
, m } , there exists some positive and finite G such that, the i -th loss f
it at each round t ∈ { 1 , . . .
, T } is G -Lipschitz continuous w.r.t.
∥·∥ , i.e., | f it ( x ) − f it ( x ′ )
| ≤ G ∥ x − x ′ ∥ .
In the convex setting, Lipschitz continuity guarantees bounded gradients, i.e., ∥∇ f it ( x ) ∥ ∗ ≤ G for any t ∈ { 1 , . .
, T } , i ∈ { 1 , . . .
, m } , x ∈ X .
We can now derive a general regret bound for OMMD.
Theorem 1.
Assume the step size η satisfies 4 V T G 2 T ≤ η ≤ 4 V T G 2 .
Then OMMD-I or OMMD-II with weights λ t at each round t attains the following regret
[Equation]
Remark.
The above theorem shows the theoretical superiority of OMMD-II over OMMD-I or naive linear scalarization that always uses fixed composition weights λ t ≡ λ at each round.
In particular, with the proper choice of α = FG 2 T V T , OMMD-II adaptively selects λ t to minimize the term ∥∇ F t ( x t )
λ t ∥ 22 + α ∥ λ t − λ t − 1 ∥ 1 .
This term will become larger with any other choice of λ t .
Finally, specific to OMMD-II, we have the following regret bound.
Note that, this bound actually relies on the assumption that Ω(1) ≤ V T ≤ o ( T ) , which is implicitly assumed in other works for dynamic online learning (Besbes et al., 2015; Yang et al., 2016; Campolongo and Orabona, 2021).
We also extend the bound to arbitrary value of V T ≥
(see Corollary 2 in the appendix).
Corollary 1.
With η = 2 G ( γDV T GT ) 1 / 3 and α = 8 FG 2 T 2 V T , OMMD-II achieves the following regret
[Equation]
E XPERIMENTS
In this section, we conduct extensive simulation and real-world experiments to evaluate the effectiveness of our proposed algorithm.
In our experiments, we mainly compare our proposed OMMD-II with two baseline algorithms: (i) linear optimal (lin-opt) performs single-objective online learning using the linearized loss λ ⊤ F at each round t , where the linearization weights λ is fixed and decided by a grid search on ∆ m ; note that, it is equivalent to using a fixed composition weight λ t
= λ in
OMMD.
(ii) min-norm extends the famous min-norm method (D´esid´eri, 2012; Sener and Koltun, 2018) to the multi-objective online setting, which has been described as OMMD-I in Algorithm 1.
5. S IMULATION E XPERIMENTS : T RACKING THE P ARETO F RONT
In the simulation setup, the goal is to track two points ξ 1 t , ξ 2 t moving on the plane R 2 .
The two points cycle along a circle with a ratius of 1, i.e., C = { ξ ∈ R 2 | ∥ ξ ∥ 2 = 1 } , namely, for each i ∈ { 1 , 2 } , ξ it =
(cos θ it , sin θ it ) is determined by some angle θ it .
For each i ∈ { 1 , 2 } , we set a positive integer P i as the rotating period, which is unknown to the learner.
The two points are initialized by θ 11 =
and θ 21 = π/ 2 , and iteratively generated as follows: at each round t , for each i ∈ { 1 , 2 } , the adversary independently samples an angle δ it from a Gaussian distribution N (2 π/P i , 1 / √ P i ) , then generates the next point ξ it +1 via θ it +
=
θ it − δ it .
Note that, we have E θ it
+
=
θ i 1 + 2 πt/P i , which implies that in average ξ it rotates clockwise uniformly along the circle C , with a period of P i .
At each round t , the learner generates a decision x t from X = { x ∈ R 2 | ∥ x ∥ 2 ≤ 2 } , which is a L 2norm ball with a radius of 2.
Then it acquires the positions of ξ 1 t , ξ 2 t and suffer two losses
f it ( x t ) =
∥ x t − ξ it ∥ 2 / 2 for i ∈ { 1 , 2 } .
In this problem, the Pareto set of the vectoral loss f t = ( f 1 t , f 2 t ) at each round t is exactly the line segment linking ξ 1 t and ξ 2 t , i.e., X ∗ t = { λξ 1 t + (1 − λ )
ξ 2 t | λ ∈ [0 , 1] } .
For any decision x t ∈ X , its PSG exactly equals to the squared distance between x t and the line segment.
The setup and PSG measurement are summarized in Figure 1 (left plot).
[Figure] [Figure]
In our experiments
[Figure]
we set T = 10 , 000 .
To simulate the adversarial dynamic change, we further set P 1 = 10 , P 2 = 20 for the first T 1 = 3 , 000 rounds, and P 1 = 20 , P 2 = 10 for the remaining
T 2 = 7 , 000 rounds.
As our goal is the track the Pareto front, we adopt the average PSG, namely (cid:80) t ∈
[ T ] ∆ t ( x t ) /T , as the performance metric.
Accompanying the lin-opt baseline, we also consider its two variants: lin-1 which sets λ as the optimal weights for the first T 1 rounds, and lin-2 which set λ to be optimal regarding the last T 2 rounds.
The learning rates η in all algorithms and the regularization parameter α in OMMD-II are set as what the corresponding theories suggest.
Figure 1 (right plot) shows the performance of all the examined algorithms.
From the results, we observe that OMMD-II achieves the lowest PSG, showing its ability to track the Pareto front; meanwhile, min-norm appears very unstable in the dynamic setting, even worse than linear scalarization.
In particular, by comparing OMMD-II and lin-1 , we find that linear scalarization with carefully selected weights may attain performance comparable to our algorithm during some certain phase; however, in the dynamic setting where the pattern drifts over time, the performance of linear scalarization may become very unstable or even drop severely, while our algorithm is much more robust.
5. O NLINE C ONVEX E XPERIMENTS : A N A PPLICATION TO A DAPTIVE R EGULARIZATION
In many real-world online applications, regularization is often applied to avoid overfitting.
A most common way is to add a regularization term r ( x ) to the loss f t ( x ) in each round, and optimize the regularized loss f t ( x ) + σr ( x ) instead (McMahan, 2011).
The strength σ of regularization is often treated as a hyperparameter that needs to be decided carefully beforehand, e.g., via a grid search.
The formalism of multi-objective online learning provides an another way to realize regularization.
Since r ( x ) often measures the complexity of x , it can be viewed as the second objective to be optimized.
Specifically, by constructing a vectoral loss F t ( x ) =
( f t ( x ) , r ( x )) in each round, we can cast the regularized online learning into a two-objective online optimization problem.
Compared to the previous approach with a fixed strength σ , in our approach, the strength is implied by the weights in gradient composition, namely σ t = λ 2 t /λ 1 t , which is adaptive at each round t .
We run experiments on two online benchmark datasets.
(i) protein : A bioinformatics dataset for protein type classification (Wang, 2002), which has 17 thousand instances with 357 features.
(ii) covtype : A biological dataset collected from a non-stationary environment, whose goal is to predict the cover type of forests at a particular location (Blackard and Dean, 1999), which has 50 thousand instances with 54 features.
For both classification tasks, we use the logistic loss as the first objective, and the squared L 2 -norm of model parameters (i.e., the L 2 -regularizer) as the second objective.
In the experiments, we adopt L 2 -norm balls as the decision set and set the diameter K = 10 .
For
OMMD-II, the parameter α is simply set as 0 . 2 .
For fixed regularization, the strength σ = λ 2 t /λ 1 t is determined by a grid search over ( λ 1 t , λ 2 t ) ∈ ∆ 2 , and we denote this method as lin-opt .
Moreover, for both OMMD-II and lin-opt , learning rates η are decided via a grid search over { 0 . 1 , 0 . 2 , . . .
Since the ultimate goal of regularization is to enhance predictive performance, we adopt the average of cumulative loss, namely (cid:80) t ∈
[ T ] l t ( x t ) /T
where l t ( x t ) is the classification loss at round t , as the performance metric.
The performance of both algorithms are reported in Figure 2.
The results show that OMMD-II attains lower loss than lin-opt in all the examined tasks, which shows the superiority of adaptive regularization using our online MOO technique over fixed regularization.
[Figure]
E7DVN/7UDLQLQJ/RVV F7DVN/7HVW/RVV G7DVN5$YHUDJH&XPXODWLYH/RVV
[Figure]
O NLINE N ON -C ONVEX E XPERIMENTS : M ULTI -T ASK D EEP L EARNING
Finally, we evaluate OMMD-II in the online non-convex setting.
We choose to use the MultiMNIST dataset (Sabour et al., 2017), which is a multi-objective version of the famous MNIST dataset for image classification and commonly used in deep multi-task learning (Sener and Koltun, 2018;
Lin et al., 2019).
In MultiMNIST, each sample is constructed by putting a random image at the top-left and another image at the bottom-right.
Our goal to classify the digit on the top-left ( task L ) and to classify the digit on the bottom-right ( task R ) at the same time.
We follow Sener and Koltun (2018)
’s setup and adopt the LeNet architecture.
For linear scalarization, we consider two choices of weights, namely (0 . 5 , 0 . 5) and (0 .
For all the examined algorithms, the learning rates η are selected via a grid search over { 0 . 0001 , 0 . 001 , 0 . 01 , 0 . 1 } .
For
OMMD-II, the parameter α is set according to our theory.
Note that, in online experiments, the sample arrives one after another in a sequential manner, which is different from stochastic optimization where sample batches are randomly sampled from the whole training set (Sener and Koltun, 2018).
Figure 3 compares the average cumulative loss, training loss and test loss of all the examined algorithms for both tasks.
Note that, the first metric is typically used in online experiments and the last two are commonly used in stochastic experiments (Reddi et al., 2018).
The results show that
OMMD-II outperforms OMMD-I ( min-norm ) and linear scalarization ( lin ) in all metrics, which shows that our proposed algorithm is also effective in the online non-convex setting.
C ONCLUSIONS
In this paper, we conduct a systematic study of multi-objective optimization in the online setting.
We first formulate the framework of Multi-Objective Online Convex Optimization.
Then we devise the
Online Mirror Multiple Descent algorithm, which is the first gradient-based multi-objective online learning algorithm and has a special design when tailoring multiple gradient algorithm to online learning, namely regularized min-norm solver.
Theoretically, we provide the first paradigm of regret analysis for multi-objective online convex optimization.
We finally conduct extensive experiments to demonstrate the effectiveness of our proposed algorithm.
Future works may include developing a framework based on the Hypervolume metric, or giving an analysis of multi-objective static regret.
E THICS S TATEMENT
As a study on a general learning problem, our work will not incur ethical issues by itself.
However, ethical issues may arise if our learning method is improperly applied to some application fields - just as any other general learning method if it is misused.
R EPRODUCIBILITY S TATEMENT
For every theoretical statement in our paper (including proposition, lemma, theorem and corollary) we give a detailed proof in Appendix C.
The codes to reproduce our empirical results are provided in the supplementary materials
A PPENDIX
The appendix is organized as follows.
Appendix A provides more details of OMMD, including a discussion on the lazy version of mirror descent and how to efficiently compute the composition weights for the regularized min-norm solver.
Appendix B supplements the theoretical results with more general regret bounds for arbitrary temporal variability V T ∈ [0 , ∞ ) ; such a general setting is often not considered in a typical analysis of dynamic online learning (Besbes et al., 2015; Zhang et al., 2018), but we believe that adding such general bounds will make the analysis more selfcontained.
Appendix C provides detailed proofs of all theoretical claims in this work, which have been omitted in our main paper due to the space limit.
A M ORE D ETAILS OF T HE A LGORITHM
In this section, we provide more details to help better understand the design of OMMD.
First, we note that, our proposed OMMD is actually based on the agile version of online mirror descent (Hazan, 2019), where the updated model directly moves to its projecting point onto the decision set at each round.
However, we can easily make an analogy and devise a lazy version of
OMMD.
Specifically, we only need to use a lazy projection operation in the mirror descent step with the composite gradient instead of the agile projection operation (line 7 in Algorithm 1).
Note that, the analysis of the lazy version is very similar to that of the agile version (Hazan, 2019).
Next, we show that, when calculating the weights λ t for gradient composition, our regularized minnorm solver only need very light computation, just as the original min-norm solver in stochastic optimization (Sener and Koltun, 2018; Lin et al., 2019).
Specifically, similar to (Sener and Koltun, 2018), we first consider the setting of two objectives, namely m = 2 .
In this case, when λ, λ t − 1 ∈ ∆ 2 , the L1 regularizer ∥ λ − λ t − 1 ∥ 1 equals to 2 | λ 1 − λ 1 t − 1 | .
Then the optimization problem on λ at round t reduces to (since there are only two objectives, the superscripts of λ 1 and λ 1 t − 1 are omitted)
[Equation]
Interestingly, we find that, the above problem also has an closed-form solution as below.
Proposition 2.
Set λ L =
( g ⊤ 2 ( g 2 − g 1 )+ α ) /
∥ g 2 − g 1 ∥ 2 , and λ R =
( g ⊤ 2 ( g 2 − g 1 ) − α ) /
∥ g 2 − g 1 ∥ 2 .
Then the above optimization problem has a closed-form solution, i.e.,
[Equation]
Proof.
We solve the following two quadratic sub-problems respectively, namely,
[Equation]
as well as
[Equation]
The former problem produces λ = max { min { λ L , λ t − 1 } , 0 } , and the latter one gives λ = max { min { λ R , 1 } , λ t − 1 } .
Therefore, the solution to the original optimization problem can then be derived by comparing the optimal values of the two sub-problems, which gives the desired solution in the proposition.
■
Now that we have derived the closed-form solution to the regularized min-norm solver with any two gradients, in priciple, we can apply Sener and Koltun (2018)’s technique to efficiently compute the solution to the solver with more than two gradients.
Specifically, we can adopt the Frank-Wolfe method, namely, iteratively choose a pair of gradients and update the composition weights with these two gradients.
B M ORE D ETAILS OF T HEORETICAL A NALYSIS
In this section, we provide more details and highlight technical lemmas in theoretical analysis, which are omitted in our main paper due to space limitation.
B. A M ORE D ETAILED R EGRET B OUND FOR OMMD-II
In Corollary 1, we only give the order of OMMD-II’s regret bound w.r.t.
the core factors m, T and
V T .
In the following theorem, we supplement a more detailed regret bound for OMMD-II which explicates the dependency of other factors on the regret.
The proof is presented in Appendix C.
Theorem 2.
Set η = 2 G ( γDV T GT ) 1 / 3 and α = 8 FG 2 T 2 V T .
Then OMMD-II attains the following regret
[Equation]
Note that, as we have discussed in our main paper, just like its simplified version Corollary 1, the above theorem is derived under the assumption of ”regular” temporal variability, namely Ω(1) ≤
V T ≤ o ( T ) , which is implicitly assumed in other works for dynamic online learning (Besbes et al., 2015; Yang et al., 2016; Campolongo and Orabona, 2021).
To give a more general analysis, in the following subsection, we also provide strict regret bounds that are valid for arbitrary V T ∈ [0 , ∞ ) .
B. R EGRET B OUNDS FOR OMMD-II U NDER A RBITRARY T EMPORAL V ARIABILITY
We here provide a more strict regret bound for OMMD-II under arbitrary temporal variability V T ∈ [0 , ∞ ) .
Specifically, we first give a detailed regret bound in analogy to Theorem 2.
The proof of this theorem is given in Appendix C.
Theorem 3.
In OMMD-II, set η = min { max { 2 G ( γDV T GT ) 1 / 3 , 4 V T G 2 T } , 4 V T G 2 } and α = 8 FG 2 T 2 V T .
Then it attains the regret bound
[Equation]
The above bound can be rewritten into a simpler form, if we are only interested in its order w.r.t.
m, T and V T , just as in Corollary 1.
Corollary 2.
In OMMD-II, set η = min { max { 2 G ( γDV T GT ) 1 / 3 , 4 V T G 2 T } , 4 V T G 2 } and α = 8 FG 2 T 2 V T .
Then it attains the following regret
R MOD ( T ) ≤ max { O ( V T 1 / 3 T 2 / 3 ) , O ( V T ) , O ( T 1 / 2 ) } .
C O MITTED P ROOFS
In this section, we provide the detailed proofs of Proposition 1, Lemma 1, Theorem 1 and Corollary 1 in our main paper as well as Theorem 2, Theorem 3 and Corollary 2 in the appendix.
C. P ROOF OF P ROPOSITION
Proof.
We first analyze the PSG measurement ∆ t ( x t ) at each round t ∈ { 1 , . . .
, T } .
Specifically, for any comparator x ∈ X , we first define the pair-wise suboptimality gap between decisions x t and x , namely,
[Equation]
ϵ
Then ∆ t ( x t ) can be evaluated via the pair-wise suboptimality gap as ∆ t ( x t ) =
sup x ∈P X ( F t )
δ t ( x t , x ) .
We then focus on the pair-wise gap δ t ( x t ; x ) w.r.t.
any Pareto optimal decision x ∈ X ∗ t ≡ P X ( F t ) .
Since x is a Pareto optimal decision of F t , from the definition of Pareto optimality, there must exist some i ∈ { 1 , . .
, m } such that f it ( x t ) − f it ( x ) ≥ 0 .
The pair-wise suboptimality gap has an equivalent expression, namely,
[Equation]
where ( l ) + = max { l, 0 } , l ∈ R is the truncation operator.
Denote U m = { e k | 1 ≤ k ≤ m } as the set of all unit vector in R m , then we equivalently have
[Equation]
Note that, we here slightly abuse the truncation operator ( l ) + to allow l to be a vector in R m , which represents the vector whose i -th coordinate equals to max { l i , 0 } for any i ∈ { 1 , . . .
, m } .
Now the calculation of δ t ( x t ; x ) becomes a minimization problem over λ ∈ U m .
Since U m is a discrete set, we can apply a linear relaxation trick.
Specifically, we now turn to minimize the quantity p ( λ ) =
λ ⊤ ( F t ( x t ) − F t ( x ))
+ over the convex curvature of U m , which is exactly the probability simplex ∆ m = { λ ∈ R m | λ ≻ 0 , ∥ λ ∥ 1 = 1 } .
Note that, U m contains all vertexes of ∆ m .
Since inf λ ∈ ∆ m p ( λ ) is a linear optimization problem, the minimal point λ ∗ t ⊤ must be a vertex of the simplex, i.e., λ ∗ t ⊤ ∈
U m .
Thus the relaxed problem is equivalent to the original problem, namely,
[Equation]
For now, we have transformed the calculation of pair-wise suboptimality gap δ t ( x t ; x ) into a optimization problem of finding the minimal linear scalarization of ( F t ( x t ) − F t ( x )) + .
Hence, the PSG at each round t can be expressed as
[Equation]
In the above expression, the existence of truncation operator ( · ) + will incur irregularity (i.e., nonlinearity) when we try to optimize the loss F t .
Suprisingly, we find that such operator can be dropped when we compute ∆ t ( x t ) w.r.t.
the Pareto set X ∗ t , as shown in the following.
Lemma 2.
Let X ∗ t be the Pareto set of F t : X → R m .
Then for any x t ∈ X , it holds that
[Equation]
Proof.
Define the alternative ”gap” metric without the truncation operator as δ ′ t ( x t ; x ) =
inf λ ∈ ∆ m λ ⊤ ( F t ( x t ) − F t ( x )) .
Moreover, define the supremum of δ ′ t ( x t ; x ) over x ∈ X ∗ t as ∆ ′ t ( x t ) = sup x ∈X ∗ t δ ′ t ( x t ; x ) .
Then from the definition of truncation operator ( · ) + , we have δ (
x t ; x ) ≥ δ ′ ( x t ; x ) and ∆ t ( x t ) ≥ ∆ ′
t ( x t ) .
It then suffices to prove that, for any given x t ∈ X , there exists some certain x ∗ t ∈ X ∗ t such that the value of δ ′ ( x t ; x ∗ t ) can be as large as ∆ t ( x t ) .
Indeed, if this is the case, then ∆ t ( x t ) ≤ ∆ ′
t ( x t ) , and hence the two quantities ∆ t ( x t ) and ∆ ′ t ( x t ) are equal.
We consider the following two cases:
(i) x t is already a Pareto optimal point of F t , i
e., x t ∈ X ∗ t .
Then from the definition of PSG, we directly have ∆ t ( x t ) = 0 .
Notice that δ ′ ( x t ; x t ) = 0 , and hence ∆ t ( x t ) = sup x ∈X ∗ t δ ′ ( x t ; x ) ≥ δ ′ ( x t ; x t ) = 0 .
Consequently, the relation ∆ t ( x t ) ≤ ∆ ′ t ( x t ) holds in this case.
(ii) x t is not a Pareto optimal point of F t ,
i
e., x t / ∈
X ∗ t .
Then we have ∆ t ( x t ) > 0 .
Set ϵ
=
∆ t ( x t ) , and denote x ∗ t ∈ arg max x ∈X ∗ t δ t ( x t ; x ) , then δ t ( x t ; x ∗ t ) = ϵ > 0 .
Therefore, from the definition of δ t ( x t ; x ∗ t ) we know that, for any i ∈ { 1 , . . .
, m } , we have f it ( x t ) − f it
( x ∗ t ) ≥ ϵ .
Thus all entries of F t ( x t ) − F t ( x ∗ ) are positive, and we have ( F t ( x t ) − F t ( x ∗ ))
+ = F t ( x t ) − F t ( x ∗ ) .
Consequently, we have ∆ ′ t ( x t ) =
sup x ∈X ∗ t δ ′ t ( x t ; x ) ≥ δ ′ ( x t ; x ∗ t ) =
δ (
x t
; x
∗
t ) =
∆
t ( x t
■
From the above lemma, the PSG measurement at round t has an equivalent form as
[Equation]
and correspondingly, the multi-objective dynamic regret becomes
[Equation]
Since the time horizon T is finite, we can first swap the summation over t and the supremum over x , then swap the summation over t and the infimum over λ .
Then the multi-objective dynamic regret further equals to
[Equation]
which proves the proposition.
■
C. P ROOF OF L EMMA
Proof.
For OMMD whose the composition weights are λ t at round t , before analyzing its regret bound, we first introduce two useful lemmas.
Lemma 3.
For OMD with stepsize η and loss λ t ⊤ F t ( x ) , we have the following recursion
[Equation]
(6) for any t ∈ { 1 , . . .
, T } .
Proof.
Our proof is similar to the analysis of OMD in the single-objective setting (Srebro et al., 2011; Cesa-Bianchi et al., 2012).
Specifically, fix f t = λ t ⊤ F t and g t = λ t ⊤ F t ( x t ) .
From the convexity of f t , we have
[Equation]
From the first-order optimal condition of x t +1 , for any x ′ ∈ X , we have
[Equation]
We set x ′ = x ∗ t in the above inequality
[Equation]
and consequently derive
[Equation]
Recall the definition of Bregman divergence B R .
We can check that (also see (Beck and Teboulle, 2003))
[Equation]
Since R is 1-strongly convex
[Equation]
we have B R ( x t +1 , x t ) ≥ ∥ x t +1 − x t ∥ 2 / 2 .
Hence
[Equation]
Moreover, from the Cauchy-Schwartz inequality we have
[Equation]
Combining the above two inequalities, we can prove the lemma.
■
Lemma 4.
For an arbitrary comparator sequence u 1 , . . .
, u T ∈ X , we have
[Equation]
Proof.
Summing the inequality in Lemma 3 over t ∈ { 1 , . .
, T } , we have
[Equation]
Recall the assumption that B R ( x, z ) − B R ( y, z ) ≤ γ ∥ x − y ∥ , ∀ x, y, z ∈ X .
We then obtain
T (cid:88)
[Equation]
[Equation]
which proves the lemma.
■
We can now return to prove Lemma 1.
Notice that, for an arbitrary comparator sequence u 1 , . .
, u T ∈ X , the Pareto tracking regret can be decomposed as T (cid:88) t =
[Equation]
term
[Equation]
We now instantiate the comparator sequence to be a piece-wise stationary sequence, i.e., { u 1 , . .
, u T } =

[Equation]
  , which starts with w ∗ 1 and only changes for every ∆ steps ( ∆ is an integer such that ∆ ≤ T ).
More specifically, for any i ∈ { 1 , . . . , ⌈ T/ ∆ ⌉} , denote p i
= ( i − 1)∆ + 1 and q i = i ∆ , and then
I i =
[ p i , q i ] is exactly
the i -th stationary piece of the comparator sequence.
For any piece i ∈ { 1 , . . .
, ⌈ T/ ∆ ⌉} , we set all u t , t ∈ I i to be the best fixed decision x
⋆ I i regarding the cumulative linearized losses during interval
I i , i.e., u t ≡ x ⋆ I i =
arg min x ∈X (cid:80) t ∈I i λ t ⊤ F t ( x ) , for any t ∈
I
i , i ∈ { 1 , . .
. , ⌈ T/ ∆ ⌉} .
Then we can apply Lemma 4 to such comparator sequence and bound the term A as
[Equation]
Notice that, the second term B measures the difference between the cumulative linearized loss of the best decisions x
∗I i regarding each interval
I i and that of the comparators { x ∗ t } .
To analyze this term, we consider such difference B i restricted to any interval
I i , i ∈ { 1 , . . . , ⌈ T/ ∆ ⌉} :
[Equation]
t
[Equation]
λ
[Equation]
F t
[Equation]
x ∗
[Equation]
Recall our definition of u t and x ∗ t , then we have (cid:80) t ∈I i λ t ⊤ F t ( u t ) ≤ (cid:80) t ∈I
i λ t ⊤ F t ( x ∗ p i ) and λ p i ⊤
F p i ( x ∗ p i ) ≤ λ p i ⊤ F p i ( x ∗ t ) .
Hence we further have
[Equation]
t ∈I i λ t ⊤ F t ( x ∗ t ) .
Moreover, for any t ∈ I i , x ∈ X , we have
[Equation]
Recall that I i has at most ∆ elements, we further have
[Equation]
Hence the term B can be bounded as
[Equation]
From the definition of p i and q i
[Equation]
we further have
[Equation]
Combining the above two inequalities on terms A and B
[Equation]
we finally derive
[Equation]
Assume that the diameter of the decision domain X is upper bounded by some D > 0 ,
then ∥ u
I i − u
I i +1 ∥ ≤ D and B R ( u I 1 , x 1 ) ≤ γ ∥ u
I 1 − x 1 ∥ ≤ γD .
Hence we have
[Equation]
Further assume that each loss function F t is coordinate-wise G -Lipschitz continuous, i.e., | f it ( y ) − f it ( x ) | ≤ G ∥ y − x ∥ , ∀ x, y ∈ X , i ∈ { 1 , . . .
, m } .
Without loss of generality, we suppose 0 ∈ X and f it (0)
= 0 , ∀ t ∈ { 1 , . . .
, T } , i ∈ { 1 , . . .
, m } .
Then for any x ∈ X , | f it ( x ) | ≤ GD , and consequently
[Equation]
It then remains to tackle the term (cid:80) T −
[Equation]
=1 sup x ∈X | λ ⊤ t ( F t ( x ) − F t +1 ( x )) | .
To this end, we introduce the following lemma.
Lemma 5.
For OMMD with composition weights λ t ∈ ∆ m at each round t , it holds that
[Equation]
1 (cid:88) t
= sup x ∈X | λ t ⊤ ( F t ( x ) − F t +1 ( x ))
| ≤ 2 F ( T − 1)
T − 1 (cid:88) t =
∥ λ t − λ t +1 ∥ 1 + V T .
Proof.
Denote λ
= (cid:80) Tk =1 λ k T .
We then decompose (cid:80) T − 1 t =1 sup x ∈X | λ
[Equation]
⊤ (
[Equation]
− F
[Equation]
+
[Equation]
x
[Equation]
) | as
T
[Equation]
[Equation]
=
[Equation]
Since we have assumed that | f it ( x ) | ≤ F for any x ∈ X , t ∈ { 1 , . . .
, T } , i ∈ { 1 , . . .
, m } , it holds that | f it ( x ) − f it +1 ( x ) | ≤ 2 F . Consequently, we can bound the term A as term A =
[Equation]
≤
T − 1 (cid:88) t
= m (cid:88) i = | λ it −
[Equation]
λ
[Equation]
| ≤
T − 1 (cid:88) t
= m (cid:88) i = | λ it − ( λ ) i | · 2 F =
T − 1 (cid:88) t = 2 F ∥ λ t − λ ∥ 1 , where λ it and ( λ ) i represents the i -th entry of λ t and λ , respectively.
Also notice that for any t ∈ { 1 , . .
, T } ,
[Equation]
≤
[Equation]
T
[Equation]
T
[Equation]
T
[Equation]
(cid:88) s =
∥ λ s −
λ s +1 ∥ 1 .
Therefore, we have term A ≤
T − 1 (cid:88) t = 2 F
T − 1 (cid:88) s =
∥ λ s −
λ s +1 ∥ 1 = 2 F
( T − 1)
T − 1 (cid:88) t =
∥ λ t − λ t +1 ∥ 1 .
As for term B, we have term B =
T − 1 (cid:88) t
= sup x ∈X | λ ⊤ ( F t ( x ) − F
[Equation]
where the last inequality is derived from the assumption of temporal variability, and the last equation comes from the fact that (cid:80) mi =1 ( λ ) i = 1 T (cid:80) mi =1 (cid:80)
Ts =1 λ is = 1 T (cid:80)
Ts =1 (cid:80) mi =
λ it = 1 .
Combining the above bounds for term A and term B, we finally prove the lemma.
■
Plugging the above terms into the above bound for R MOD ( T ) , and replace the quantity ∆ ∈ { 1 , . .
, T } by δ , we have
[Equation]
for any δ ∈ { 1 , . . .
, T } .
Note that (cid:80)
Tt =
1 sup x ∈X ( | λ t ⊤ ( F t ( x ) − F t +1 ( x )) | + | ( λ t − λ t +1 )
⊤ F t +1 ( x ) | ) =
(cid:80) T − 1 t =1 sup x ∈X ( | λ t ⊤ ( F t ( x ) − F t +1 ( x )) | + | ( λ t − λ t +1 )
⊤ F t +1 ( x ) | ) , because the regret does not depend on F T +1 , λ T +1 .
We thus prove the lemma.
■
C. P ROOF OF T HEOREM
Proof.
This theorem can be directly derived from Lemma 1.
Specifically, when 4 V T G 2 T ≤ η ≤ 4 V T G 2 , we can set δ = ηG 2 T V T , which satisfies 1 ≤ δ ≤ T .
Plugging it into Lemma 1 and rearranging the inequality, we can directly derive the theorem.
■
C. P ROOF OF T HEOREM
Proof.
Since this theorem is a special case of its following Theorem 3 when Ω(1) ≤ V T ≤ o ( T ) , it can be proved as we derive Theorem 3 in the following subsection.
In fact, as we assume Ω(1) ≤ V T ≤ o ( T ) , in the following derivation of Theorem 3, we are always in the case of (i).
In addition, when Ω(1) ≤ V T ≤ o ( T ) in Theorem 3 we exactly have
[Equation]
Hence this theorem can be directly derived from Theorem 3.
■
C. P ROOF OF T HEOREM
Proof.
Denote η 0 = 2 G ( γDV T GT ) 1 / 3 .
We consider the following three cases:
(i)
When 4 V T G 2 T ≤ η 0 ≤ 4 V T G 2 , we can directly apply the above lemma.
(ii) When η 0 < 4 V T G 2 T , or equivalently V T > ( γD 8 ) 1 / 2 GT , we have η = 4 V T G 2 T .
Set δ = 1 in Lemma 1,
then it
can
be verified
that
[Equation]
T .
(iii) When η 0 >
V
T
G
2 , or equivalently
V T < (
γD
8 T ) 1 /
G ,
we have η
=
V T G
Set δ = T in Lemma 1,
then it can be verified that
[Equation]
Combining (i)-(iii)
[Equation]
we prove the theorem.
■
C. P ROOF OF C OROLLARY
Proof.
Since this corollary is a special case of its following Corollary 2 when Ω(1) ≤ V T ≤ o ( T ) , it can be directly derived from Corollary 2.
Specifically, since V T ≥ Ω(1) , we have V 1 / 3 T
T 2 / 3 ≥ O ( T 1 / 2 ) .
Moreover, since V T ≤ o ( T ) , we have V 1 / 3 T 2 / 3 ≥ o
( V T ) .
Therefore, the dominating term in the bound of Corollary 2 is V 1 / 3 T 2 / 3 , which proves the corollary.
■
C. P ROOF OF C OROLLARY
Proof.
We start from the general bound derived in Theorem 3.
Specifically, in the first regret term, since λ t is selected to minimize α ∥ λ t − λ t − 1 ∥ 1 + ∥∇ F t ( x t ) λ t ∥ ∗ 2 at each step t , we further have min
[Equation]
Plugging it into the bound in Theorem 3 directly proves the corollary.
■
D M ORE DETAILS IN M ULTI -O BJECTIVE S TATIC R EGRET
As we have discussed before, the multi-objective static regret cannot be formulated using most existing discrepancy metrics, since these metrics always give non-negative measurements, and the static regret based on any non-negative metric will fail to reduce to the standard static regret R S ( T ) in the single-objective setting.
In this section, we give a possible form of multi-objective static regret and provide an analysis for it.
The static regret is enlightened by the equivalent form of dynamic regret derived in Proposition 1.
Recall that in Proposition 1, the comparator x ∗ t at each round t is selected from the Pareto set X ∗ t of the instantaneous loss F t ; in addition, the weights λ ∗ t at each round t is generated separately.
To derive a static version, we can use a fixed comparator x ∗ from the Pareto set X ∗ of the cumulative loss (cid:80) t F t and fixed weights λ ∗ at all rounds.
Now the static regret takes
[Equation]
, where X ∗ =
P X ( (cid:80) Tt =1 F t ) .
Note that when m = 1 , the probabilistic simplex ∆ m reduces to a single point { 1 } , and the Pareto optimal set X ∗ coincides with the optimal set of the cumulative scalar loss (cid:80)
Tt =1 F t , i.e., X ∗ =
arg min x ∈X F t ( x ) .
Hence the multi-objective static regret takes R MOS ( T ) =
sup x ∈X ∗ ( (cid:80) Tt =1 F t ( x t ) − (cid:80)
Tt =1 F t ( x ∗ ))
=
(cid:80) Tt =1 F t ( x t )
− min x ∈X (cid:80)
Tt =
1 F t ( x ) , which exactly reduces to the standard static regret R S ( T ) in the single-objective setting.
Surprisingly, with proper choices of η and α , our proposed OMMD-II algorithm can still achieve a sublinear bound w.r.t.
T for the static regret, as shown in the following theorem.
Note that the derived regret bound for OMMD-II is tight w.r.t.
T , since it matches the lower bound O ( √ T ) of the standard static regret in the single-objective setting (Hazan, 2019).
Theorem 4.
OMMD-II with composition weights λ t attains the following static regret
[Equation]
By setting η = √ 2 D G √ T and α = 4 FTη , the regret bound reduces to O ( √ T ) , which is sublinear w.r.t.
T .
Proof.
We start from the definition of R MOS ( T ) .
Specifically, for any λ ∈ ∆ m and λ 1 . . .
, λ T ∈ ∆ m , it holds that
[Equation]
To tackle the second term in the above inequality
[Equation]
we set u 1 = u 2 = . . .
=
u T = x ∗ in Lemma 4, which results in
[Equation]
To analyze the static regret for OMMD-II
[Equation]
we relate the first term (cid:80)
Tt =1 ∥ λ − λ t ∥ 1 with the Lregularization in the regularized min-norm solver at each round.
Specifically, we denote the average weights as ¯ λ = (cid:80) Tt =
1 λ t /T
and set λ = ¯ λ in the first term.
Then for any t ∈ { 1 , . . .
, T } , we have
[Equation]
Consequently, the static regret can be bounded as
[Equation]
which proves the full static regret bound.
Now we prove the reduced bound.
From the full bound
[Equation]
we equivalently have
[Equation]
We now set λ = 4 FTη in OMMD-II, and specify λ t to be the composition weights generated by the algorithm at round t .
Then we know that λ t ∈ arg min λ ∈ ∆ t ∥∇ F t ( x t ) λ
∥ 2 ∗ + 4 FTη ∥ λ − λ t − 1 ∥ 1 .
In particular, we have ∥∇ F t ( x t ) λ t ∥ 2 ∗ + 4 FTη ∥
λ t − λ t −
[Equation]
≤ ∥∇ F t ( x t ) λ t − 1 ∥ 2 ∗ .
Therefore, it holds that
[Equation]
Utilize Assumption 1 and Assumption 3, and set η = √ 2 D G √ T , then we have
[Equation]
which proves the reduced bound.
■ 𝑥 !
=
𝐹 !
𝑥 !
= 0, 𝒳 !
∗ 𝒫 !
∗ 𝑥 "#$ = (−1,−1) 𝐹 ! 𝑥 "#$ 𝑥 %&’( = (−1,0) 𝐹 ! 𝑥 %&’( 𝐹 ! 𝑥 ) ,𝑥 * = 2𝑥 ) − 𝑥 *,2𝑥 * − 𝑥 )
Note that the newly introduced static regret is no longer based on PSG.
It can be understood as induced from a new discrepancy metric δ ( x t ; x ∗ , F t , λ ∗ )
=
( λ ∗ ) ⊤ ( F t ( x t ) − F t ( x ∗ )) , where λ ∗ ∈ ∆ m .
We thus have R MOS ( T ) =
sup x ∗ ∈X ∗ inf λ ∗ ∈
∆ m (cid:80) Tt =1 δ ( x t ; x ∗ , F t , λ ∗ ) .
Such a metric compares the generated decision x t at each round with the fixed comparator x ∗ ∈
X ∗ .
Hence, in particular, it is able to produce a negative value when x t dominates x ∗ regarding the instantaneous loss F t , making it a general extension of PSG.
Although this property is desired in the definition of static regret, it is rarely utilized in the discrepancy metrics in multi-objective optimization.
Thus it looks a bit strange as a Pareto suboptimality metric and hence its physical meaning needs to be justified further.
Besides, there are undoubtedly many other possible ways to define such metrics.
Therefore, regarding the multi-objective static regret, much work remains to be done in the future.
We hope our initial attempt paves the way for future research.