Multi-Objective Online Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
This paper presents a systematic study of multi-objective online learning.
We firstformulate the framework of Multi-Objective Online Convex Optimization, whichencompasses two novel multi-objective regret definitions.
The regret definitionsbuild upon an equivalent transformation of the multi-objective dynamic regretbased on the commonly used Pareto suboptimality gap metric in zero-order multiobjective bandits, making it amenable to be optimized via first-order iterativemethods.
To motivate the algorithm design, we give an explicit example in whichequipping OMD with the vanilla min-norm solver for gradient composition willincur a linear regret, which shows that only regularizing the iterates, as in singleobjective online learning, is not enough to guarantee sublinear regrets in the multiobjective setting.
To resolve this issue, we propose a novel min-regularized-normsolver that regularizes the composite weights.
Combining min-regularized-normwith OMD results in the Doubly Regularized Online Mirror Multiple Descentalgorithm.
We further derive both the static and dynamic regret bounds for theproposed algorithm, each of which matches the corresponding optimal bound in thesingle-objective setting.
Extensive experiments on both simulation and real-worlddatasets verify the effectiveness of the proposed algorithm.
Introduction
Traditional optimization methods for machine learning are usually designed to optimize a singleobjective.
However, in many real-world applications, we are often required to optimize multiple correlated objectives concurrently.
For example, in autonomous driving [12, 20], the self-drivingvehicles need to solve multiple tasks such as self-localization and object identification at the sametime.
In online advertising [21, 22], advertisers need to determine the exposure of items to differentusers to maximize both the Click-Through Rate (CTR) and the Post-Click Conversion Rate (CVR).
In many multi-objective scenarios, the objectives may conflict with each other [15].
Hence, there maynot exist any single solution that optimizes all the objectives simultaneously.
For example, in onlineadvertising, merely optimizing CTR or CVR will degrade the performance of the other [21, 22].
Multi-objective optimization (MOO) [23, 6] is concerned with optimizing multiple conflictingobjectives simultaneously.
It seeks Pareto optimality, where no single objective can be improvedwithout hurting the performance of the others.
Many different methods for MOO have been proposed,including evolutionary methods [26, 39], scalarization methods [9], and gradient-based iterativemethods [7].
Recently, the Multiple Gradient Descent Algorithm (MGDA) and its variants have beenintroduced to the training of multi-task deep neural networks and achieved great empirical success[29], making them regain a significant amount of research interest [17, 33, 18].
These methodscompute a composite gradient based on the gradient information of all the individual objectivesand then apply the composite gradient to update the model parameters.
The composite weights aredetermined by a min-norm solver [7] which yields a common descent direction of all the objectives.
However, compared to the increasingly wide application prospect, the gradient-based iterativealgorithms are relatively understudied, especially for the online learning setting.
Multi-objectiveonline learning is of essential importance due to reasons in two folds.
First, due to the data explosion inmany real-world scenarios such as web applications, making in-time predictions requires performingonline learning.
Second, the theoretical investigation of multi-objective online learning will lay a solidfoundation for the design of new optimizers for multi-task deep neural networks.
This is analogous tothe single-objective setting, where nearly all the optimizers for training DNNs are initially analyzedin the online setting, such as AdaGrad [8], Adam [16], and AMSGrad [28].
In this paper, we give a systematic study of multi-objective online learning.
To begin with, weformulate the framework of Multi-Objective Online Convex Optimization (MO-OCO).
The firstmajor challenge is the lack of regret definitions in the multi-objective setting.
To tackle this challenge,we need appropriate discrepancy metrics that can be used in the regret definitions, which evaluate thegap between any two vector losses by producing scalar values.
Intuitively, the Pareto suboptimalitygap (PSG) metric, which is frequently used in zero-order multi-objective bandits [30, 19], is a verypromising candidate.
It can yield scalarized distances from any vector loss to a given comparator set.
We can thus define the multi-objective regret by simply plugging in PSG as the discrepancy metric.
However, as a metric designed purely from the geometric view, PSG is intrinsically difficult to beoptimized directly via gradient-based iterative methods.
To resolve this problem, for the PSG-basedmulti-objective dynamic regret, we derive its equivalent unconstrained max-min form via a highlynon-trivial transformation.
This form is intuitive to the design of first-order multi-objective onlinealgorithms, indicating that we should select a convex combination of the gradients at each round.
Unfortunately, for the PSG-based static variant, such an equivalence does not exist.
To remedy thisissue, we make extensions of the dynamic variant by fixing the comparator set and the compositeweights, which yields an appropriate definition of the multi-objective static regret.
Based on the MO-OCO framework, we develop a novel multi-objective online algorithm termed
Doubly Regularized Online Mirror Multiple Descent.
The key module of the algorithm is the gradientcomposition scheme, which calculates a composite gradient in the form of a convex combination ofthe gradients of all objectives.
Intuitively, the most direct way to determine the composite weights isto apply the min-norm solver [7] commonly used in offline multi-objective optimization.
However,directly applying min-norm is not workable in the online setting.
Specifically, the composite weightsin min-norm are only determined by the gradients at the current round.
In the online setting, sincethe gradients can be adversarial, they may result in undesired composite weights, further producinga composite gradient that reversely optimizes the loss.
To rigorously verify this point, we give ashowcase in which equipping OMD with vanilla min-norm even incurs a linear regret, showing thatonly regularizing the iterate, as in OMD, is not enough to guarantee sublinear regrets in the multiobjective setting.
To fix this issue, we devise a novel min-regularized-norm solver with an explicitregularization on composite weights.
Equipping it with OMD results in our proposed algorithm.
We then conduct the theoretical analysis for our proposed algorithm.
We derive a multi-objective staticregret bound O ( √ T ) and a multi-objective dynamic regret bound O
( V 1 / 3 T
T 2 / 3 ) for DR-OMMD.
Both bounds match the optimal bounds in the single-objective setting [11, 34].
Our analysis alsoshows that DR-OMMD attains a lower regret than linearization with fixed composite weights.
To evaluate the effectiveness of DR-OMMD, we conduct extensive experiments on both simulationdatasets and real-world datasets.
We first elaborate simulation experiments, in which we findthat DR-OMMD attains lower regret than vanilla min-norm and linearization, which verifies thesuperiority of the min-regularized-norm solver.
We then realize adaptive regularization via multiobjective optimization on real-world datasets, and find that adaptive regularization with DR-OMMDsignificantly outperforms fixed regularization with linearization.
In summary, in this paper, we give the first systematic study of multi-objective online learning, whichencompasses a novel framework, a new algorithm, and corresponding non-trivial theoretical analysis.
We believe that this work paves the way for future research on more advanced multiple-objectiveoptimization algorithms, which may inspire the design of new optimizers for multi-task deep learning.
Preliminaries
In this section, we briefly review the necessary background knowledge of online convex optimizationand multi-objective optimization.
2.1 Online Convex Optimization
Online Convex Optimization (OCO) [38, 11] is the most commonly adopted framework fordesigning online learning algorithms.
It can be viewed as a structured repeated game between alearner and an adversary.
At each round t ∈ { 1 , . . .
, T } , the learner is required to generate a decisionx t from a convex compact set
X ⊂ R n .
Then the adversary replies the learner with a convex functionf t : X → R and the learner suffers the loss f t ( x t ) .
The goal of the learner is to minimize the regretwith respect to the best fixed decision in hindsight, i.e.,
[Equation]
Note that the above regret is the static regret [10], which compares the learner’s cumulative losswith that of a fixed decision.
There is another version of regret, namely the dynamic regret [10, 34],which compares the learner’s cumulative loss with that of a sequence of local optimal decisions, i.e.,
[Equation]
Any meaningful regret is required to be sublinear in T , i.e., lim T →∞ R S/D ( T ) /T
= 0 , which impliesthat when T is large enough, the learner can perform as well as the best fixed decision in hindsight(for static regret) or the local optimal decision at each round (for dynamic regret).
Online Mirror Descent (OMD) [11] is a classic first-order online learning algorithm.
At each roundt ∈ { 1 , . . .
, T } , OMD yields its decision using the following formula
[Equation]
where η is the step size, R : X → R is the regularization function, and B R ( x, x ′ )
=
R ( x ) − R ( x ′ ) −⟨∇
R ( x ′ ) , x − x ′ ⟩ is the Bregman divergence induced from R .
As a meta-algorithm, by instantiatingdifferent regularization functions, OMD can induce two important algorithms, i.e., Online Gradient
Descent [38, 13] and Online Exponentiated Gradient [11].
2.2 Multi-Objective Optimization
Multiple-objective optimization (MOO) is concerned with solving the problems of optimizingmultiple objectives simultaneously [39, 29].
In general, since different objectives may conflict witheach other, there is no single solution that can optimize all the objectives at the same time.
Instead,
MOO seeks to find solutions that achieve Pareto optimality.
Next, we exposit Pareto optimality andrelated definitions more formally using a vector-valued loss H
= ( h 1 , . . .
, h m ) ⊤ as objectives, wherem ≥ 2 and h
i : K → R , i ∈ { 1 , . . .
, m } , K ⊂ R , is the i -th loss function.
Definition 2.1 ( Pareto optimality ) .
(a)
For any two solutions x, x ′ ∈ K , we say that x dominatesx ′ , denoted as x ≺ x ′ or x ′ ≻
x , if h i ( x ) ≤ h
i ( x ′ ) for all i , and there exists one i such thath i ( x ) <
h i ( x ′ ) ; otherwise, we say that x does not dominate x ′ , denoted as x ⊀ x ′ or x ′ ⊁ x .(b)
A solution x ∗ ∈ K is called Pareto optimal if it is not dominated by any other solution in K .
There may exist multiple Pareto optimal solutions.
For example, it is easy to show that the optimizerof any single objective, i.e., x ∗ i ∈ arg min x ∈K h
i ( x ) , i ∈ { 1 , . . .
, m } , is Pareto optimal.
Different
Pareto optimal solutions reflect different trade-offs among the objectives [17].
Definition 2.2 ( Pareto front ) .
(a) All Pareto optimal solutions form the Pareto set
P K ( H ) .
(b) The image of P K ( H ) constitutes the Pareto front, denoted as P ( H )
= { H ( x ) | x ∈ P K ( H ) } .
Now that we have established the notion of optimality in MOO, we proceed to introduce the metricsthat measure the discrepancy of an arbitrary solution x ∈ K from being optimal.
Recall that, in thesingle-objective setting with merely one loss function h : Q → R , where Q ⊂ R , for any z ∈ Q ,the loss gap h
( z ) − min z ′′ ∈Q h ( z ′′ ) is directly the discrepancy measure.
However, in MOO withmore than one loss, for any x ∈ K , the loss gap H ( x ) − H ( x ′′ ) , where x ′′ ∈ P K ( H ) , is a vector.
Intuitionally, the desired discrepancy metric shall scalarize the vector-valued loss gap and yieldthe value 0 for any Pareto optimal solution.
In general, there are two commonly used discrepancymetrics in MOO, i.e. Pareto suboptimality gap (PSG) [30] and Hypervolume (HV) [4].
As HV is avolume-based metric, it is more difficult to optimize or analyze via iterative algorithms [36].
Hencein this paper, we adopt PSG, which has been extensively used in multi-objective bandits [30, 19].
Definition 2.3 ( Pareto suboptimality gap ) .
For any x ∈ K , the Pareto suboptimality gap to a givencomparator set K ∗ ⊂ K , denoted as ∆( x ; K ∗ , H ) , is defined as the minimal scalar ϵ ≥ 0 that needsto be subtracted from all entries of H ( x ) , such that H ( x ) − ϵ 1 is not dominated by any point in K ∗ ,where 1 denotes the all-one vector in R m , i.e.,
[Equation]
Clearly, PSG is a distance-based discrepancy metric that motivated from a purely geometric viewpoint.
In practice, the comparator set K ∗ is often set to be the Pareto set P K ( H ) [30].
Then for any x ∈ K ,its PSG is always non-negative and equals to zero if and only if x ∈ P K ( H ) .
Multiple Gradient Descent Algorithm (MGDA) is an offline first-order algorithm for MOO [9, 7].
At each iteration l ∈ { 1 , . . .
, L } ( L is the number of iterations), it first computes the gradient ∇ h
i ( x l )for each objective i ∈ { 1 , . . .
, m } , then derive the composite gradient g compl = (cid:80) mi =1 λ il ∇ h
i ( x l ) asthe convex combination of these multiple gradients; it applies g compl to execute the gradient descentstep to update the decision, i.e., x l +1 = x
l − ηg compl , where η is the step size.
The core part of
MGDA is the module that determines the composite weights λ l = ( λ 1 l , . . .
, λ ml ) , which is given as
[Equation]
where S m = { λ ∈ R m | (cid:80) mi
=
λ i = 1 , λ i ≥ 0 , i ∈ { 1 , . .
. , m }} denotes the probabilistic simplex in 150 R m .
This is a min-norm solver which finds the weights in the simplex that yields the minimum L 2norm of the composite gradient.
Thus MGDA is also called the min-norm method.
Existing works[7, 29] have shown that MGDA is guaranteed to decrease all the objectives simultaneously until itreaches a Pareto optimal decision (under the convex setting where all h i are convex functions).
Multi-Objective Online Convex Optimization
In this section, we formally formulate the framework of multi-objective optimization in the onlinesetting, termed Multi-Objective Online Convex Optimization (MO-OCO).
Framework overview.
We tailor the famous online convex optimization (OCO) framework to themulti-objective setting, which can be viewed as a repeated game between an online learner and theadversarial environment.
At each round t ∈ { 1 , . . .
, T } , the learner generates a decision x t from agiven convex compact decision set X ⊂ R n .
Then the adversary replies the decision with a vectorloss function F t ( x ) :
X → R m , where its i -th component f it ( x ) :
X → R belongs to the i -thobjective, and the learner suffers the loss F t ( x t ) ∈ R m .
The goal of the learner is to generate asequence of decisions { x t } Tt =1 so that the cumulative loss (cid:80) Tt =1 F t ( x t ) can be optimized.
Recall that, in the single-objective setting, the performance metric R ( T ) =
(cid:80) Tt =
( f t ( x t ) − f t ( z t )) ,i.e., the regret, compares the actual decisions x t with some comparator z t ∈ X at each round t .
Forthe static regret, all z t are identically set as the fixed optimal decision x ∗ w.r.t.
all losses in hindsight,i.e., z t ≡ x ∗ ∈ arg min x ∈X (cid:80) Tt =1 f t ( x ) .
For the dynamic regret, each z t is selected as the optimaldecision x ∗ t w.r.t.
the instantaneous loss f t at that round, i.e., z t = x ∗ t ∈ arg min x ∈X f t ( x ) .
In analogy, we can define the multi-objective regret as R ( T ) =
(cid:80) Tt =1 ∆ t , where each ∆ t comparesthe actual decisions x t with some comparator z t ∈ X .
However, in general, no single decision canoptimize all the objectives at the same time.
Hence, it is natural to compare x t with a group of Paretooptimal decisions, which constitute a comparator set C t ⊂ X .
To measure the discrepancy between x tand C t , we further introduce the Pareto suboptimality gap (PSG) [30] ∆( x t ; C t , F t ) .
Then the multiobjective regret can be defined as R ( T ) =
(cid:80) Tt =1 ∆( x t ; C t , F t ) .
Now we can formulate the static orthe dynamic variant by specifying the comparator set C t at each round.
Specifically, by setting all C t tobe the Pareto set X ∗ of the cumulative loss (cid:80)
Tt =1 F t , we formulate the multi-objective static regret
R MOS ( T ) =
(cid:80) Tt =1 ∆( x t ; X ∗ , F t ) .
By setting each C t to be the Pareto set X ∗ t of the instantaneousloss F t , we formulate the multi-objective dynamic regret R MOD ( T ) =
(cid:80) Tt =1 ∆( x t ; X ∗ t , F t ) .
Recall that PSG is a zero-order metric motivated in a purely geometric sense, namely, its calculationneeds to solve a constrained optimization problem with an unknown boundary f it ( x ′′ ) , ∀ x ′′ ∈ C t .
Hence, it is not straightforward to design a first-order algorithm to optimize PSG, not to mentionthe regret analysis.
To motivate algorithm design and analysis, we investigate the two variants inmore detail.
We begin with the dynamic variant, since we find that it has an equivalent form, which isintuitive and has a strong implication on the design of effective online multiple gradient algorithms.
An equivalent form of the dynamic regret.
Surprisingly, the multi-objective dynamic regret R MODcan be transformed into an unconstrained max-min form.
The derivation utilizes Pareto optimality of
X ∗ t and is highly non-trivial, which is deferred to the appendix due to the space limit.
Proposition 3.1.
The multi-objective dynamic regret has an equivalent form, i.e.,
[Equation]
Remark.
(i)
The above form can be understood as a variant of the standard dynamic regret regarding{ λ ∗ t ⊤ F t } Tt =1 , whereas λ ∗ t are unknown to the learner.
This provides an intuition that we can generate weights λ t ∈ S m at each round and optimize { λ t F t } Tt =1 via single-objective techniques.
Forfirst-order algorithms, it is equivalent to selecting a convex combination of individual gradients andthen applying the composite gradient to model update.
Undoubtedly, how to generate the weights λ tneeds some careful designs, which will be explicated later in the algorithm section.(ii)
When m = 1 , we have S m = { 1 } and X ∗ t = arg min x ∈X F t ( x ) .
Hence R MOD ( T ) = 196 (cid:80)
Tt =
( F t ( x t ) − min x ∈X F t ( x )) , which is exactly the single-objective dynamic regret R D ( T ) .
An alternative form of the static regret.
Unfortunately, for R MOS , the above equivalence formdoes not exist.
Here is the reason.
In R MOS , the comparator set X ∗ is the Pareto set of the cumulativeloss (cid:80)
Tt =1 F t rather than the instantaneous loss F t .
Hence, at some specific round t , the decisionx t may Pareto dominate all points in X ∗ w.r.t.
the instantaneous F t , and we would expect themetric ∆ t to be negative.
However, PSG (or other commonly used metrics such as Hypervolume)always yields non-negative values, so the induced R MOS is not aligned with R S .
For example, whenm = 1 , we have R MOS ( T ) =
sup x ∗ ∈X ∗ (cid:80)
Tt =1 max { F t ( x t ) − F t ( x ∗ ) , 0 } , which can be much looserthan the static regret R S ( T ) =
sup x ∗ ∈X ∗ (cid:80) Tt =1 ( F t ( x t ) − F t ( x ∗ )) .
Hence the analysis of R MOS isintrinsically complex if we use existing discrepancy metrics that always yield non-negative values.
Enlightened by Proposition 3.1, we can formulate the static regret in a different way, i.e., by modifyingthe equivalent form of dynamic regret.
Recall that in Proposition 3.1, at each round t , the comparatorx ∗ t is selected from the Pareto set X ∗ t of the instantaneous loss F t , and the weights λ ∗ t are generatedfrom S m .
To formulate the static variant, we can use a fixed comparator x ∗ from the Pareto set X ∗ ofthe cumulative loss (cid:80) t F t and fixed weights λ ∗ ∈ S m at all rounds.
Now the static variant takes
[Equation]
) − (cid:88) T t =1 F t ( x ∗ )) .
Remark.
(i) R MOS ( T ) has a clear physical meaning that optimizing it will impose the cumulativeloss (cid:80)
Tt =1 F t ( x t ) to reach the Pareto front P ∗ .
See more details in Appendix C.(ii)
When m = 1 , S m = { 1 } and X ∗ reduces to arg min x ∈X (cid:80) Tt =
1 F t ( x ) .
Therein R MOS ( T ) = 214 (cid:80)
Tt =1 F t ( x t ) − min x ∗ ∈X ∗ (cid:80)
Tt =1 F t ( x ∗ ) , which reduces to the single-objective static regret R S ( T ) .
Online Mirror Multiple Descent
In this section, we present the Online Mirror Multiple Descent (OMMD) algorithm, the protocol ofwhich is given in Algorithm 1.
At each round t , the learner first computes the gradient of the lossregarding each objective, then determines the composite weights of all these gradients, and finallyapplies the composite gradient to the online mirror descent step.
4.1 Vanilla Min-Norm May Incur Linear Regrets
The core module of OMMD is the composition of multiple gradients.
For simplicity, we representthe gradients at round t in a matrix form ∇ F t ( x t ) =
[ ∇ f 1 t ( x t ) , . .
. , ∇ f mt ( x t )]
∈ R n × m .
Then the
1: Input time tion do 4:
Compute
[Equation]
Determine the weights for the gradient composition via min-regularized-norm
[Equation]
Compute the composite
Perform online mirror descent using g
[Equation]
9: end for composite gradient is given as g t = ∇ F t ( x t ) λ t , where λ t is the composite weights.
As illustrated in
Preliminary, the min-norm method in MGDA [7, 29] is a classic method to determine the compositeweights in the offline setting, which results in a common descent direction that can descend all thelosses simultaneously.
Thus, it is tempting to consider applying it to the online setting.
However, directly applying the min-norm method to the online setting is not workable, which mayeven incur linear regrets of the resulting algorithms.
The rationale is as follows.
In the vanillamin-norm method, the composite weights λ t are determined solely by the gradients ∇ F t ( x t ) at thecurrent round t , hence they are very sensitive to the instantaneous loss F t .
In the online setting,the losses at each round can be adversarially chosen, and thus the corresponding gradients can beadversarial.
These adversarial gradients may result in undesired composite weights, which mayfurther produce a composite gradient that even deteriorates the next prediction.
In the following,we provide a problem instance in which min-norm incurs a linear regret.
We extend OMD to themulti-objective setting, where the composite weights are directly yielded by min-norm [11].
Problem instance.
We consider a two-objective problem.
The decision domain is X =
{ ( u, v ) |u + v ≤ 12 , v − u ≤ 12 , v ≥ 0 } and the loss function at each round is
[Equation]
where a = ( − 2 , − 1) ,
b = (0 , 1) , c
= (2 , − 1) .
For simplicity, we first analyze the case where thetotal time horizon T is an even number.
Then we can compute the Pareto set of the cumulativeloss (cid:80)
Tt =1 F t , i.e., X ∗ = { ( u, 0) | − 12 ≤ u ≤ 12 } , which locates at the x -axis.
For conciseness ofanalysis, we instantiate OMD with L2-regularization, which results in the simple OGD algorithm[24].
We start at an arbitrary point x 1 = ( u 1 , v 1 ) ∈
X satisfying v 1 > 0 .
At each round t , supposethe decision x t = ( u t , v t ) ∈ X , then the gradients of each objective w.r.t.
x t can be calculated as
[Equation]
Since 0 ≤ v t ≤ 12 , we observe that the second entry of either gradient alternates between positiveand negative.
By using min-norm, the composite weights λ t can be computed as
[Equation]
We observe that both entries of composite weights alternative between above 1 2 and below 1 2 , and∥ λ t +1 − λ t ∥ 1 ≥ 1 .
Recall that ∥ λ t ∥ 1 = 1 , hence the composite weights at two consecutive roundschange radically.
The resulting composite gradient takes
[Equation]
The fluctuating composite weights mix with the positive and negative second entries of gradients,making the second entry of g compt always negative, i.e., − u t + v t − 1 < 0 and − u t − v t − 1 < 0 .
Hence g compt actually drives x t away from the Pareto set X ∗ that coincides with the x -axis.
Thisessentially reversely optimizes the loss, hence increases the regret.
In fact, we can prove that it evenincurs a linear regret 2 .
Due to the lack of space, we leave the proof of linear regret when T is an oddnumber in the appendix.
The above results of the problem instance are summarized as follows.
Proposition 4.1.
For OMD equipped with vanilla min-norm, there exists a multi-objective onlineconvex optimization problem, in which the resulting algorithm incurs a linear regret.
Remark.
Stability is a basic requirement to guarantee meaningful regrets in online learning [25].
In the single-objective setting, directly regularizing the iterate x t (e.g., OMD) is already enough.
However, as shown in the above analysis, only regularizing x t is not enough to attain sublinear regretsin the multi-objective setting, since there is another source of instability, i.e., the composite weights,that affects the direction of the composite gradient.
Therefore, in multi-objective online learning,besides regularizing the iterates, we also need to explicitly regularize the composite weights.
4.2 Doubly Regularized Online Mirror Multiple Descent
Enlightened by the design of regularization in FTRL [25], we consider the regularizer r ( λ, λ 0 ) , whereλ 0 is the pre-defined composite weight that may reflect the user preference.
This results in a newsolver called min-regularized-norm , i.e.,
[Equation]
where α is the strength of regularization.
Equipping OMD with the new solver, we derive theproposed online algorithm.
Note that beyond the regularization on the iterate x t that is intrinsic inonline learning, there is another regularization on the composite weights λ t in min-regularized norm.
Both regularizations are fundamental and they together ensure the stability in the multi-objectiveonline setting.
Hence we call the algorithm Doubly Regularized OMMD (DR-OMMD).
In principle, r can take various forms such as L 1 -norm, L 2 -norm and KL divergence etc.
Herewe adopt L 1 -norm since it aligns well with the simplex constraint of λ .
Min-regularized-normcan be computed very efficiently, since it has a closed-form solution when m = 2 .
Specifically,suppose the gradients at round t are g 1 t and g 2 t .
Set γ L =
( g ⊤ 2 ( g 2 − g 1 ) − α ) /
∥ g 2 − g 1 ∥ 2 andγ R =
( g ⊤ 2 ( g 2 − g 1 ) + α ) /
∥ g 2 − g 1 ∥ 2 .
Given any λ 0 = ( γ 0 , 1 − γ 0 ) ∈ S 2 , we can compute thecomposite weights λ t as ( γ t , 1 − γ t ) where
[Equation]
where γ ′′ t = max { min { γ 0 , γ R } , γ L } .
In addition, when m > 2 , since the feasible region S m is a simplex, we can introduce a Frank-Wolfesolver [14] to compute the composite weights.
See the protocol and more details in Appendix D.
Compared to vanilla min-norm, the composite weights in min-regularized-norm are not fully determined by the adversarial gradients.
The resulting relative stability of composite weights make thecomposite gradients more robust to the adversarial environment.
In the following, we give a generalanalysis and prove that DR-OMMD indeed guarantees sublinear regrets.
4.3 Analysis
We now analyze the static regret and the dynamic regret of DR-OMMD.
Our analysis is based on thefollowing commonly used assumptions [13, 11].
Assumption 4.2 ( Bregman divergence ) .
The regularization function R is 1 -strongly convex.
Inaddition, the Bregman divergence is γ -Lipschitz continuous, i.e., B R ( x, z ) − B R ( y, z )
≤ γ ∥ x −y ∥ , ∀ x, y, z ∈ dom R, where dom R is the domain of R and satisfies X ⊂ dom R ⊂ R n .
Assumption 4.3 ( Lipschitz continuity ) .
For each i ∈ { 1 , . . .
, m } , there exists some positive andfinite G such that, the i -th loss f
it at each round t ∈ { 1 , . . .
, T } is G -Lipschitz continuous w.r.t.
∥ · ∥ ,i.e., | f it ( x ) − f it ( x ′ )
| ≤ G ∥ x − x ′ ∥ .
Note that in the convex setting, this assumption leads to boundedgradients, i.e., ∥∇ f it ( x ) ∥ ∗ ≤ G for any t ∈ { 1 , . .
, T } , i ∈ { 1 , . . .
, m } , x ∈ X .
We first provide the static regret bound.
The proof is left to the appendix due to the lack of space.
Theorem 4.4.
Suppose the diameter of X is bounded by D .
Assume F t is bounded, i.e., | f it ( x ) | ≤
F, ∀ x ∈ X , t ∈ { 1 , . . .
, T } , i ∈ { 1 , . . .
, m } .
For any λ 0 ∈ S m , DR-OMMD attains
[Equation]
Remark.
(i) Linearization with weights λ 0 ∈ S m can be viewed as single-objective optimizationon scalar loss λ ⊤ 0 F t , whose gradient is g t = ∇ F t ( x t ) λ 0 .
Hence we can directly borrow the tightbound of OMD (Theorem 6.8 in [27]) and derive a bound 1 η B R ( x ∗ , x 1 ) + (cid:80)
Tt =
1 η t 2 ∥∇ F t ( x t ) λ
∥ 22for linearization.
In comparison, if we set α = 4 F/η in DR-OMMD, then from the formulation ofλ t , the bound becomes 1 η B R ( x ∗ , x 1 ) + η 2 (cid:80) Tt
=1 min λ ∈S m {∥∇ F t ( x t ) λ
∥ 2 + α ∥ λ − λ
∥ 1 } , which issmaller than that of linearization.
Note that the lower regret of DR-OMMD compared to linearizationis also empirically verified in our experiments (see Figure 1).(ii)
When η = √ 2 γD G √ T , α = 4 Fη , the bound is in the order of O ( √ T ) .
It matches the optimal staticsingle-objective regret bound w.r.t.
T [11] (see more details in Appendix E).
Then we turn to the dynamic regret.
Our analysis relies on an additional assumption [2, 32, 5].
Assumption 4.5 ( Temporal variability ) .
For each i ∈ { 1 , . . .
, m } , there exists some positive andfinite V T such that (cid:80) T − 1 t =1 sup x ∈X | f it ( x ) − f it +1 ( x ) | ≤ V T .
Theorem 4.6.
Assume the step size satisfies 4 V T G 2 T ≤ η ≤ 4 V T G 2 .
Then under all the above assumptions,for any preference λ 0 ∈ S m , OMMD with min-regularized-norm attains
[Equation]
Remark.
When η = 2 G ( γDV T GT ) 1 / 3 , α = 8 FG 2 T V T , the bound is in the order of O
( T 2 / 3 V 1 / 3 T ) , matchingthe best attainable single-objective dynamic regret bound [2, 35] (see more details in Appendix E).
Experiments
In this section, we conduct extensive experiments to evaluate the effectiveness of DR-OMMD.
Weconsider two baselines: (i) linearization performs single-objective online learning on the linearizedloss λ ⊤ 0 F t at each round t , where the weights λ 0 ∈ S m are given beforehand; note that it is equivalentto computing composite gradients with fixed weights λ t ≡ λ 0 .
(ii) min-norm equips OMD withvanilla min-norm [7] for gradient composition.
5.1 Simulation Experiments: Tracking the Pareto Front
As summarized in Figure 1 (a), the goal is to track two points ξ 1 t , ξ 2 t cycling along a circle C = { ξ ∈ 321 R| ∥ ξ ∥ 2 = 1 } .
For each i ∈ { 1 , 2 } , ξ it =
(cos θ it , sin θ it ) is determined by some angle θ it .
We seta positive integer P i as the rotating period of ξ it , which is unknown to the learner.
The two pointsare initialized by θ 11 =
and θ 21 = π/ 2 and move as follows: at each round t , for each i ∈ { 1 , 2 } ,the adversary independently samples an angle δ it from a Gaussian distribution N (2 π/P i , 1 / √ P i ) ,then moves the i -th point to ξ it +
= (cos θ it +1 , sin θ it +1 )
where θ it +
=
θ it − δ it .
Note that E θ it +
=
θ i 1 + 2 πt/P i , hence in average ξ it rotates clockwise with a period of P i .
At each round t , the learnergenerates a decision x t from a L 2 -norm ball X = { x ∈ R 2 | ∥ x ∥ 2 ≤ 2 } .
Then it acquires ξ 1 t , ξ 2 t andsuffer the losses f it ( x t ) =
∥ x t − ξ it ∥ 22 / 2 , i ∈ { 1 , 2 } .
In this problem, the Pareto set of F t = ( f 1 t , f 2 t )is exactly the line segment between ξ 1 t and ξ 2 t , i.e., X ∗ t
= { λξ 1 t
+ (1 − λ ) ξ 2 t | λ ∈ [0 , 1] } .
At eachround t , PSG measures the squared distance between x t and X ∗ t .
We run T = 10 , 000 rounds.
To simulate the pattern drift, we set P 1 = 10 , P 2 = 20 at the first
T 1 = 3 , 000 rounds, and P 1 = 20 , P 2 = 10 at the last T 2 = 7 , 000 rounds.
For linearization,the weights λ
=
( λ 10 , 1 − λ 10 ) are decided via a grid search λ 10 ∈ { 0 , 0 .
1 , ..., 1 } ; we considerthree variants: lin-1 uses the optimal λ 0 for the first T 1 rounds, lin-2 uses the optimal λ 0 for thelast T 2 rounds, and lin-opt uses the optimal λ 0 for all T rounds.
For DR-OMMD, for fairness of
[Figure]
E/HDUQLQJ&XUYH 1 t
[Figure]
2 t
[Figure]
1 t
[Figure]
[Figure]
[Figure] [Figure]
comparison we use the same λ 0 of lin-opt . The learning rates η in all algorithms and the parameterα in DR-OMMD follow the corresponding theories (e.g., Theorem 4.6).
In this experiment, sincethe loss functions are manually designed, the value of V T can be directly calculated.
Note that insome scenarios where V T is unknown, we can conduct a grid search and utilize a meta-algorithm tohandle the unknown V T [37, 1], similar to the single-objective setting.
From the results in Figure 1(b), we find that DR-OMMD achieves the lowest PSG, showing its ability to track the Pareto front;meanwhile, min-norm appears very unstable in the online setting, even worse than linearization.
5.2 Convex Experiments: Adaptive Regularization via Multi-Objective Optimization
In many real-world online scenarios, regularization is often adopted to avoid overfitting.
A standardway is to add a term r ( x ) to the loss f t ( x ) at each round and optimize the regularized loss f t ( x ) +σr ( x ) [24], where σ is treated as a hyperparameter that needs to be fixed beforehand.
The formalismof multi-objective online learning provides a novel way to realize regularization.
Since r ( x ) measuresthe complexity of x , it can be regarded as the second objective alongside the primary goal f t ( x ) .
Wecan construct a vector loss F t ( x ) =
( f t ( x ) , r ( x )) at each round and thereby cast regularized onlinelearning into a bi-objective online optimization problem.
Compared to fixed regularization, the newapproach effectively chooses the regularization strength σ t = λ 2 t /λ 1 t in an adaptive way.
We use two large-scale online benchmark datasets.
(i) protein is a bioinformatics dataset for proteintype classification [31], which has 17 thousand instances with 357 features.
(ii) covtype is a biologicaldataset collected from a non-stationary environment for forest cover type prediction [3], which has50 thousand instances with 54 features.
For both tasks, we set the logistic loss of classification asthe first objective, and the squared L 2 -norm of model parameters as the second objective.
Since theultimate goal of regularization is to enhance predictive performance, we adopt the average loss as theperformance metric, namely (cid:80) t ≤ T l t ( x t ) /T , where l t ( x t ) is the classification loss at round t .
We adopt a L 2 -norm ball centered at the origin with diameter K = 100 as the decision set.
Thelearning rates are decided by a grid search over { 0 . 1 , 0 . 2 , . . .
For DR-OMMD, the parameter αis simply set as 0 . 1 .
For fixed regularization, the strength σ =
(1 − λ 10 ) /λ 10 is determined by the somepreference λ 1 0 ∈ [0 , 1] , which is essentially linearization with weights λ 0 = ( λ 1 0 , 1 − λ 10 ) .
We runboth algorithms with varying initial weights λ 10 ∈ { 0 , 0 . 1 , ..., 1 } .
In Figure 2, we plot (a) their finalperformance w.r.t.
the choice of λ 0 and (b) their learning curves with desirable λ
(e.g., (0 . 1 , 0 . 9) onprotein ).
Other results are deferred to the appendix due to the lack of space.
The results show that
DR-OMMD consistently outperforms fixed regularization.
Conclusions
In this paper, we give a systematic study of multi-objective optimization in the online setting.
Wefirst formulate the framework of Multi-Objective Online Convex Optimization.
Then we devisethe Doubly Regularized Online Mirror Multiple Descent algorithm, which has a special design forgradient composition in online learning, namely min-regularized-norm.
We provide non-trivial regretbounds for DR-OMMD and conduct extensive experiments to demonstrate its effectiveness.
Limitations.
As the first step of studying multiple gradient algorithm in online learning, we conductour analysis in the convex setting.
Although it does not affect the usage in the non-convex setting(see empirical validation in Appendix F), we can give a formal non-convex analysis in the future.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work
is concerning a general problem in online learning.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix G, H, I.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] They are
included in the supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 5.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] We conduct online learning experiments,
where the
learning process is deterministic.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix E.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes] We cite the source of datasets.
(b) Did you mention the license of the assets? [Yes]
In the supplemental material
c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Our codes are provided in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
We only use publicly available benchmark datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
We only use publicly available benchmark
datasets.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]