On the Convergence of Stochastic Multi-Objective Gradient Alteration and Beyond
Anonymous Author(s)
AffiliationAddress email
Abstract
The conflicting gradients problem is the major bottleneck of the effective training ofmodels that deal with multiple objectives.
To resolve this problem, several gradientalteration techniques, such as PCGrad, MGDA, and CAGrad, are developed bydirectly altering the conflicting gradients to refined ones with fewer or even no conflicts.
However, the design and analysis of these techniques are mainly conductedunder the exact full-batch gradient setting, ignoring that they are primarily appliedwith stochastic mini-batch gradients.
In this paper, we point out that the stochasticgradient alteration algorithms may fail to converge to Pareto optimal solutions.
Wesummarize these seemingly different algorithms into a unified algorithmic framework, where the descent direction is given by the composition of the gradientsw.r.t.
the multiple objectives.
Then we provide a simple two-objective convexoptimization instance to illustrate the non-convergence issue in detail, which showsthat the non-convergence results from the determination of the composite weightssolely by the stochastic gradients.
To fix this issue, we propose a novel composite weights determination scheme that exponentially averages the past calculatedweights.
Finally, we show that the resulting new variants of stochastic gradientalteration converge to Pareto optimality under the unified framework and also giverise to improved empirical performance for PCGrad, MGDA, and CAGrad.
Introduction
Multi-objective optimization (MOO) refers to the optimization of multiple objectives simultaneously [11, 6, 5].
MOO problems appear in many real-world scenarios, e g., the online advertisingmodel [36, 37, 30, 47] aims to maximize both the Click-Through Rate (CTR) and the Post-Click
Conversion Rate (CVR); autonomous driving system [17, 35] needs to learn to solve multiple taskssuch as self-localization and object identification at the same time.
The main challenge lies in theinherent conflicts between multiple objectives that impairs the training [22, 50, 31].
For example,in online advertising, merely optimizing either CVR or CTR will degrade the performance of theother [36, 37].
Various MOO algorithms [41, 50, 31, 46, 32, 19] were proposed to deal with theconflicting problem from by altering the gradients evaluated from multi-objective losses.
Typicallymultiple-objective gradient alteration (MOGA) methods such as MGDA [41], PCGrad [50], and
CAGrad [31] adjust the multiple gradients to seek a local direction that descents all the objectives.
While these algorithms have been successfully applied in a wide range of applications and achievedsignificant performance gain, the current design and analysis are derived with exact (e.g., full-batch)objective gradients [41, 50, 31, 46, 32, 19].
However, in most real-world scenarios, evaluatingstochastic (e.g., mini-batch) gradients is much more practical from the computational cost aspect,especially for the training of deep neural networks [25, 39].
Since altering the stochastic gradients could further amplify the negative impact of the stochasticnoise, such a gap between full and stochastic gradients may cause significant convergence issues.
Inparticular, current MOGA algorithms only seek a local direction that only focuses on the multiplestochastic gradients in one step, while the common descent direction for the stochastic objectivespossibly is not the right direction to descend the true ones in expectation.
In this paper, we analyze this problem and rigorously prove that such concern indeed exists.
Surprisingly, we find that in some cases, altering the stochastic gradients yields a direction that evendegrades all the objectives in expectation.
Such an inverse direction that moves the algorithms inthe wrong direction will resist the solution from convergence.
This implies that current MOGAalgorithms are suboptimal for MOO.
In particular, we make the following main contributions.
To uniformity analyze different MOGA algorithms, we summarize them in a general framework asd k
= − (cid:80) mi
=1 λ ik ∇ f i ( x k ) , where f 1 , . . .
, f m be the objectives of interest, { λ i } mi =1 are coefficientsof the different gradients at iteration k .
Since the composite gradient d k denotes the alternated direction from the receive gradients, the composite weights { λ i } mi =1 are functions of the multiple gradients.
We provide strict equivalent transformation for typical methods as MGDA, PCGrad, CAGrad into thesumarized form.
This framework also contains recently proposed GradVac [46], IMTL-G [32], etc.
In the stochastic setting, it represents as d k = − (cid:80) mi =
λ
i g i ( x k ) , where { g i ( x k ) } mi =1 are stochasticgradients that are unbiased with the exact ones.
Under the above general framework, we elucidate how the MOGA algorithms can cause nonconvergence by providing an example of a simple stochastic convex optimization problem where
MGDA, PCGrad, and CAGrad provably do not converge to an optimal solution.
The key reason isthat the composite weights { λ i } mi =1 are calculated only from the instantaneous gradients, thus sharingthe same randomness.
This strong correlation between the composite weights and stochastic gradientscould lead to estimation bias, which forces the composite gradient in the wrong direction that evendegrades all the objectives.
Since MGDA, PCGrad, and CAGrad possess typical properties among
MOGA algorithms, our analysis easily extends to others contained in our general framework as well.
Furthermore, we point out that the previous analysis of convergence for the MGDA algorithm [34] isestablished on the unjustified assumption, where { λ i } mi =1 are Lipschitz functions w.r.t.
each gradient.
We rigorously prove that this assumption is not true for MGDA, PCGrad, and CAGrad.
The above analysis suggests that in order to achieve guaranteed convergence, the optimizationalgorithm must reduce the correlation between the composite weights and stochastic gradients.
Toresolve this issue while preserving the practical advantages of MOGA, we fix MOGA algorithmswith a principled and practical momentum mechanism that exponentially averages the compositeweights instead of just using the values computed from the instantaneous gradients.
Theoretically,momentum reduces the variance of the composite weight [4], making the correlation reduced as well.
Thus, the inverse direction problem can be alleviated, and we prove that gradient alteration algorithmscan converge to optimal with the same rate as single objective stochastic optimization by adding thismechanism.
Our approach is simple to implement and agnostic to model and optimizer and hencecan be easily plugged into various gradient alteration algorithms that follow the general framework.
Finally, we empirically verify the proposed algorithm in different multi-objective learning scenarios,including simulation and real-world deep multi-task learning tasks, and we observe consistentperformance improvement without introducing computational or memory overhead.
Preliminaries
In this section, we briefly review the basic background knowledge of multiple-objective optimization,and introduce typical algorithms as well as their convergence analysis.
2.1 Basic Concepts of Multiple-Objective Optimization
Multiple-objective optimization (MOO) is concerned with solving the problems of optimizing multipleobjective functions simultaneously [52, 41], termed asmin x
[Equation]
where m ≥ 2 denotes the number of objectives, and K ⊂ R d denotes the feasible set in a d dimensionspace.
Each objective f i : K
→ R , i ∈ { 1 , . .
. , m } is the i -th loss function.
Different from single- objective optimization where solutions x , x ′ can be order by f ( x ) ≤ f ( x ′ ) or f ( x ) ≥ f ( x ′ ) .
In
MOO, it could have two parameter vectors where one performs better for task i and the other performsbetter for task
j ̸
= i .
Therefore, Pareto optimality is defined to deal with such an incomparable case.
Definition 1 (Pareto optimality) .
For any two solutions x , x ′ ∈ K , we say that x dominates x ′ ,denoted as x ≺ x ′ or x ′ ≻
x , if f i ( x ) ≤ f i ( x ′ ) for all i , and there exists one i such thatf i ( x ) < f i ( x ′ ) ; otherwise, we say that x does not dominate x ′ , denoted as x ⊀ x ′ or x ′ ⊁ x . Asolution x ∗ ∈ K is called Pareto optimal if it is not dominated by any other solution in K .
Note that there are a set of Pareto optimal solutions, termed as Pareto set .
The goal of MOO is tofind a Pareto optimal solution, which is necessary to be Pareto critical [3].
Definition 2 (Pareto criticality) .
A solution x ∗ ∈ K is called Pareto critical if there is no directiond ∈ R d such that ∇ f i ( x ∗ )
⊤ d < 0 , i = 1 , . . .
m for all objectives.
This definition indicates that if x is not Pareto critical, such a direction d is a descent direction for Fat point x .
Optimizing through d in the local neighborhood of x is able to get a better solution thatdominates x [11].
Since Pareto criticality reflects the local property compared with Pareto optimality,it is often used as the local minimal condition for MOO with non-convex objectives [12].
We thenpresent the sufficient conditions for determining Pareto criticality and Pareto optimality, which appearas metrics to study the convergence for MOO algorithm
Denotes the probability simplex as
S m = { ( λ 1 , . . .
, λ m ) | (cid:80) mi
=
λ i = 1 , λ ∈ [0 , 1] } , we have
Proposition 1.
For MOO problem 1, (a) If there exists λ ∈ S m such that ∥ (cid:80) mi =
λ i ∇ f i ( x ∗ ) ∥
= 0 ,then x ∗ ∈ K is Pareto critical.
(b) For convex objectives
f i ( · ) , i = 1 , . . .
, m .
If there exists λ ∈ S msuch that x ∗ = arg min x ∈K λ T F ( x ) , then x ∗ is Pareto optimal 1 .(a) is derived from the an equivalent definition of Pareto critical points in [31].
(b) refers to Theorem5.13 and Lemma 5.14 in [18], it implies that the minimizer of any linearization is Pareto optimal.
2.2 Multi-Objective Gradient Alteration
Similar to single-objective optimization, MOO can be solved by running iteratively with gradientbased algorithms.
Suppose n denotes the number of iterations.
At each k = 1 , . . . , n round,the key challenge is the gradient conflicting problem, i.e., ∇ f i ( x k ) ⊤ ∇ f j ( x k ) < 0 , i ̸
=
j, i, j ∈[ m ]= { 1 , . . .
, m } ,
which leads to performance drop for an objective when optimizing another one[50], i.e. optimizing f i along the negative gradient direction −∇ f i ( x k ) will degrade f j .
MultiObjective Gradient Alteration (MOGA) algorithms aim to search for a direction d
k that is notconflicting with each gradient.
Using such a non-conflicting d k to execute the gradient descentstep to update the decision i.e., x k +
= x k + η d k where η is the step size, is shown to get betterperformance in practice [41, 50, 31].
Mathematically, we know that f i ( x k + η d k ) − f i ( x k )
≈η d ⊤ k ∇ f i ( x k ) , i = 1 , . . .
, m by the first-order Taylor approximation assuming η is small [31].
Therefore, if d ⊤ k ∇ f i ( x k ) < 0 for each objective i , then this direction is able to descent all objectivessimultaneously.
There are various ways to compute such kind of direction, and we next introducetypical MOGA methods as follows.
Multiple Gradient Descent Algorithm (MGDA) [41] directly optimizes towards the Pareto criticality by leveraging Definition 2.
In each iteration k , MGDA aims to find a direction d
k tomaximize the minimum decrease across the losses, i.e. max d ∈ R d min i ∈ [ m ] ( f i ( x k ) − f i ( x k +η d k ))
≈ − η min d ∈ R d max i ∈ [ m ] ∇ f i ( x k )
⊤ d k .
By regularizing the norm for d k on the rightside, it computes the direction by d k = arg min d ∈ R d { max i ∈ [ m ] ∇ f i ( x k )
⊤ d + 12 ∥
d ∥ 22 } .
Thissubproblem can be rewritten equivalently as the following differentiable quadratic optimizationd k , µ = arg min { d ∈ R d ,µ ∈ R } 12 ∥
d
∥
+ µ, s.t. ∇ f i ( x k )
⊤ d ≤ µ.
If µ < 0 , then ∇ f i ( x k )
⊤ d k < 0 ,which means x k is not Pareto critical from Definition 2, and d k is the direction to descent all theobjectives simultaneously [11, 12].
To simplify the optimization, such a primal problem has a dualobjective as a min-norm oracle λ k = arg min λ k ∈ S m ∥ (cid:80) m i =
λ ik ∇ f i ( x k ) ∥ , The direction is thencalculated by d k = − (cid:80) mi =
λ ik ∇ f i ( x k ) .
Projecting Conflicting Gradients (PCGrad)
[50] alters the conflicting gradients by projectingeach onto the normal plane of the other, preventing the interfering components of the gradient frombeing applied to the network.
Specifically, it initializes the projected gradients as v PC i = ∇ f i ( x k ) .
If projected gradient v i and raw gradient ∇ f j ( x k )
are conflicting, i.e., v ⊤ i ∇ f j ( x k ) < 0 , thenit iteratively updates the projected gradients by v PC i =
[Equation]
∇ f
[Equation]
( x k ) , fori, j = 1 , . .
, m until the projected gradients do not conflict with the received gradients.
Since theconflicting components are overall eliminated, it makes the direction d
k
= − 1 m (cid:80) mi =1 v PC
i satisfiesd ⊤ k ∇ f i ( x k ) < 0 for each objective i , thus optimizing all objectives simultaneously.
Conflict-Averse Gradient (CAGrad)
[31] generalizes averaging linearization and MGDA.
Specifically, it constrains search region for the common direction as a circle around the average gradient v 0 = (cid:80) mi =1 1 m ∇ f i ( x k ) with diagram c ∥ v 0 ∥ .
Specifically, the direction d k is yielded byd k = arg min d ∈R d max i ∈ [ m ] ∇ f i ( x k )
⊤ d , s.t.
∥
d − v 0 ∥ ≤ c ∥ v 0 ∥ .
It can minimize the averageloss function, while leveraging the worst local improvement of individual tasks to regularize thealgorithm trajectory by using the common descent property of MGDA.
Note that the gradient conflicting issue is not absolutely eliminated due to the new constrain, but the primal objective forcesdirection to approach the common descent direction as closed as possible.
The dual objective is
[Equation]
⊤ v 0 + c ∥ v 0 ∥ (cid:13) (cid:13) (cid:80) m i =
λ ik ∇ f i ( x k ) (cid:13) (cid:13) .
Then the direction iscalculated by d k = − (cid:0) v 0 + c ∥ v 0 ∥ (cid:0) (cid:80) mi =1 λ ik ∇ f i ( x k ) (cid:1)(cid:1) .
Convergence analysis.
MGDA has been shown to converge to an arbitrary Pareto critical/optimalpoint with the same rate as single-objective optimization [12].
A similar result has been proved with
PCGrad [50].
CAGrad does not directly optimize towards Pareto optimality, but the averaging loss1 m (cid:80) mi =1 f i ( x ) .
It has been shown to converge to the minimizer of the averaging loss when c ∈ [0 , 1) ,and an arbitrary Pareto critical/optimal point when c ≥ 1 [31].
By Proposition 1, we can infer thatthe minimizer of the averaging loss also lies in the Pareto set.
Hence, we analyze them uniformlyunder the Pareto optimality in the following analysis.
General Framework for Multi-Objective Gradient Alteration
In this section, we summarize multi-objective gradient alteration (MOGA) algorithms into a generalframework with the goal of uniformly analyzing the above algorithms.
3.1 The Framework with Full-bath Gradient
Without loss of generality, we summarize the direction search of MOGA in the following framework
[Equation]
(2) where λ k = ( λ 1 k , . . .
, λ mk ) ⊤ is a function of the multiple gradients, since λ k is obtained by asubproblem with respect to the received gradients, e.g. the dual form for MGDA and CAGrad.
Wehere consider a more general assumption on the boundness of λ k for including various MOGAmethods than the probability simplex space in MGDA.
Assumption 1.
For the general framework 2, for any k = 1 , . . . , n, i = 1 , . . .
, m , there exists a finiteconstant B ∈ R , such that 0 ≤ λ ik ≤ B, (cid:80) mi =1 λ ik ≥ 1 .
We next introduce how this framework fits with typical methods.
The dual form of MGDA directlyfollows the framework 2.
For PCGrad, we take the two-objective case as an example.
When thegradients are not conflicting, we have λ k
=
When they are conflicting, we can rewrited k = − 12 ( v PC 1 + v PC 2 ) by extending v PC 1 and v PC
[Equation]
Then λ k = − ( 12 − ∇ f 1 ( x k ) ·∇ f 2 ( x k ) 2 ∥∇ f 1 ( x k ) ∥, 12 −
[Equation] [Equation]
x
[Equation] [Equation]
2 ∥∇
[Equation] [Equation]
x
[Equation]
) , Since the more than three objectivescases should be presented as a protocol, we leave more details for the derivation in the Appendix.
The direction d k of CAGrad is essentially an aggregation of two linear combinations as shown in itsdual objective, thus also following the general framework.
Furthermore, all three algorithms satisfy
Assumption 1, as discussed in Appendix.
Remark.
The general framework also contains more recent approaches that are based on the fundamentals of the introduced methods, such as GradVac [46], IMTL-G [32].
Note that we focus on theoptimization side for multi-objective problems, thus we provide the derivation of existing fundamentaloptimizers.
A vertical line of works that focus on neural architecture, such as RotoGrad [19], are notincluded in this work.
3.2 The Framework with Stochastic Gradients
MOO can be solved with gradient alteration algorithms with full gradients with theoretical guarantees [12, 50, 31].
However, training with mini-batch gradient is much more practical for deep learningapplications [39], while MOGA with stochastic gradients is largely underexplored.
Specifically,we only obtain noisy gradient feedback G k = (cid:0) g 1 ( x k , ξ 1 k ) , . .
g m ( x k , ξ mk ) (cid:1) that is corrupted bystochastic noise ξ ik , i = 1 , . . .
, m .
Following 2, we have its stochastic version
[Equation]
Note that here λ k depends on stochastic gradients, not the exact ones.
We then make the followingstandard assumption similar to single objective setting [21, 15], which says that we have access tounbiased stochastic estimates of the gradients for all objectives.
Assumption 2.
Let { x k } nk =1 be the iterate sequence for any algorithm.
For all k = 1 , . . . , n , we haveaccess to g 1 ( x k , ξ 1 k ) , . . .
, g m ( x k , ξ mk ) which are unbiased estimates of ∇ f 1 ( x k ) , . . . , ∇ f m ( x k ) ,i.e. E ξ k (cid:2)
g i ( x k , ξ ik ) (cid:3) = ∇ f i ( x k ) , i = 1 , . .
, m. Further, each gradient variance is bounded by
E ξ k (cid:2)
∥ g i ( x k , ξ ik ) − ∇ f i ( x k ) ∥ 22 (cid:3) ≤ σ 2 , i = 1 , . . . , m .
To analyze MOGA in the stochastic setting, we state the following definition to determine a sequencegenerated by an iterative algorithm to asymptotic converge to Pareto critical and optimal.
Definition 3 (Convergence condition) .
(a) A sequence { x k } ∞ k =1 asymptotically converges to Paretocritical points if lim k →∞ E [min λ ∗ k ∈ S m ∥∇ F ( x k ) λ ∗ k ∥ ]
→ 0 .(b)
For convex multi-objective function F ( · ) .
A sequence { x k } ∞ k =1 asymptotically converges to
Pareto optimal if lim k →∞ E [max x ∗ k ∈K min λ ∗ k ∈ S m ( λ ∗ k ⊤ F ( x k ) − λ ∗ k ⊤ F ( x ∗ k ))]
→ 0 .
The definition (a) directly follows Proposition 1 (a) and the definition of Pareto critical.
Thedefinition (b) is because x ∗ k is the minimum of λ ∗ k ⊤ F ( x ) , thus is Pareto optimal (Proposition 1 (b)).
Asymptotically converging to λ ∗ k ⊤ F ( x ∗ k ) implies that x k approaches Pareto optimal.
Note that theminimization operator of λ ∗ k is from the existence condition in Proposition 1 (b).
This definition canalso be viewed as the stochastic version of metric function [43] and Pareto suboptimal gap [20].
The Non-convergence of Stochastic Multi-Objective Gradient Alteration
With the problem setup of stochastic MOGA in the previous section, we now discuss fundamentalflaws among the current gradient alteration methods that are all designed and analyzed based on fullbatch gradients.
We show that such batch algorithms fail to converge to a Pareto optimal solution evenin a simple two-objective two-dimensional convex stochastic setting.
The main issue lies in the strongdependence of λ k on the received stochastic gradients G k .
Surprisingly, we find that combining suchcorrelated λ k with the stochastic gradient yields a biased composite gradient E
[ d k ] = − E [ G k λ k ]that even conflicts with all full-batch gradients in expectation.
This inverse direction causes thealgorithms even degrade all the objectives simultaneously.
An illustrative example.
Next, we provide a simple problem instance in which MGDA, PCGrad,
CAGrad can not converge to Pareto optimal.
Consider the following two-dimensional stochasticoptimization setting over the domain K = { ( x, y ) | x ∈ [ − 1 , 1] , y ≥ 0 } , where p represents theprobability of occurrence
[Equation]
The expected function is f 1 ( x, y ) =
( x + 3)
+ y 2 + 8 , f 2 ( x, y ) =
( x − 3)
+ y 2 + 8 .
Thus, byignoring the constant term, the optimization goal is to minimize the distance towards ( − 3 , 0) and (3 , 0)simultaneously.
We can easily know that the Pareto set is the line segment { ( a, 0) | a ∈ [ − 1 , 1] } .
−∇𝑓 !
−∇𝑓 "
PCGrad MGDA
CAGrad 𝔼[𝒅] (a) Inverse Direction 0.0 0.2 0.4 0.6 0.8 1.0 Iterations × 10 5 8.
f 1 ( x , y )
MGDACR-MGDA
PCGradCR-PCGrad
CAGradCR-CAGrad (b) Loss0.0 0.2 0.4 0.
0.8 1.0 Iterations × 10 5 9.
f 2 ( x , y )
MGDACR-MGDA
PCGradCR-PCGrad
CAGradCR-CAGrad (c) Loss0.0 0.2 0.4 0.
0.8 1.0 Iterations × 10 5 0.
D i s t an c e t
o P a r e t
o S e t
MGDACR-MGDA
PCGradCR-PCGrad
CAGradCR-CAGrad (
[Figure]
Gap
In the first step, by calculating the expected gradient of the point (0 , 0 . 4) , we find that all the gradientalteration algorithms generate a vertical-up direction in expectation, as shown in Figure 1a, whichmoves the algorithm in the wrong direction away from Pareto optimal setting.
More seriously, theexpected direction
E [ d k ] = (0 , ϵ ) , ϵ > 0 is conflicting with both true gradients, suggesting that itdegrades both objectives.
Furthermore, we could prove that the inverse direction problem happens with Pareto optimal points,which resists the algorithms to approach Pareto optimal.
Using the intuition above, we show thatgradient alteration algorithms can not converge to Pareto optimal for this setting.
Theorem 1.
There is a stochastic convex optimization problem for which MGDA, PCGrad, CAGraddo not converge to the Pareto optimal solution.
We relegate all proofs to the appendix.
A few remarks are in order.
In the proof, we provide strongersupport that at least in the range { ( x, y ) | x ∈ [ − 1 , 1] , y ∈
[0 , 0 . 05] } , the expected directions forthe three typical algorithms all move the solution away from Pareto set.
This means there exists aconstant gap for gradient alteration algorithms to approach the Pareto optimal.
Note that this gap is asufficient condition to verify the non-convergence and can be much larger, as shown in Figure 1d.
Intuitively, the convergence gap is caused by the strong correlation between the composite weightsλ k and the stochastic gradients G k , i.e., λ k is solely determined by G k , which could produce aninverse direction in the above example.
When the composite weights is independent of the stochasticgradients, the expected composite gradient E [ d k ] = − E [ G k λ k ] = −∇ F ( x k ) λ k will be a positivedirection to approach the Pareto set.
Hence, methods like linear scalarization are able to attain a pointin the Pareto optimal set [14], but are deprived of the practical benefits of gradient alteration.
We would also like to emphasize that while the example of non-convergence is carefully constructedto more easily demonstrate the problems in the above three typical algorithms, such an issue alsohappens with more general Gaussian noise, as shown in Appendix.
Hence, it is not unrealistic tosuspect scenarios where such an issue can impair practical performance.
It is also reasonable toimagine that this problem is especially aggravated in high dimensional settings, where the variance ofthe gradients is more significant.
Comparison with existing convergence results.
As introduced in Section 2.2, the optimal solutionfor the multi-task objective 1 m (cid:80) mi =
f i ( x ) must be Pareto optimal.
Therefore, the above nonconvergence analysis generates well with MOGA algorithms that optimize the averaging loss, suchas CAGrad.
This result shows that the previous convergence analyses in the full-batch setting are allinvalid in the stochastic setting.
To the best of our knowledge, there is only MGDA that has convergence analysis in the stochasticsetting.
Liu et al.
[34] establishes the convergence of MGDA with stochastic gradients, but based ontwo unjustified assumptions.
The first is that the gradient variance asymptotically converges to 0.
Thesecond is that the subproblem for λ k , denoted as function
λ k
( g 1 , . .
g m ) , is Lipschitz continuouswith right to each gradient.
The first assumption needs to gradually increase the batch size to fullbatch; however, the reliance on giant batch sizes is impractical for deep learning training [4].
Thesecond one can significantly simplify the convergence analysis, but is actually not true as shown inthe following proposition.
do 4:
Update
Compute d λ
Update x η k d k .
8: end for
Proposition 2.
Subproblem λ k ( · ) is not Lipschitz continuous with right to each gradient for MGDA,i.e., there exist no finite β such that ∥ λ k ( g 1 , . .
, g i , . . .
, g m ) − λ k ( g 1 , . .
g ′ i , . .
g m )
∥ ≤ β ∥
g i −g ′
i ∥ for i
= 1 , . .
, m and gradient g 1 , . .
, g i , g ′ i , . . . , g m .
Similar results can be proved for PCGrad and CAGrad, and we relegate the proof to the appendix.
Correlation-Reduced Stochastic Multi-Objective Gradient Alteration
In this section, we develop a new principled exponential moving average mechanism on the compositeweights λ k , and provide its theoretical benefit for fixing the convergence issue for MOGA algorithms.
Our aim is to devise a new strategy with guaranteed convergence while preserving the practicalbenefits of MOGA.
Algorithmic insight.
To understand the design of such a mechanism, recall that the main issue of thenon-convergence for MOGA lies in the strong correlation between λ k and the stochastic gradients
G k , which could cause the composite gradient E [ d k ] = − E [ G k λ k ] degrading all the objectives inexpectation.
It is natural to consider reducing such correlation, which can be bounded by
Lemma 1.
Under Assumption 1, 2.
The correlation between λ k and G k can be bounded as∥ E [ G k λ k ] − E [ G k ] E [ λ k ] ∥ 22 ≤ mB 2 σ 2 V ξ k
( λ k ) , where V ξ k ( λ k ) =
E ξ k [ ∥ λ k − E ξ k
[ λ k ] ∥ 22 ] .
From Lemma 1, we know that the correlation can be reduced by reducing the variances of λ k .
Fromthis intuition, we can modify MOGA algorithms to reduce such correlation by regularizing thevariation of λ k .
Correlation-reduced MOGA (CR-MOGA) adopts a momentum mechanism withcoefficient α k on the update of composite weights as Line 5 in Algorithm 1, which is shown to beable to reduce variance as
Lemma 2.
Under Assumption 1, λ k variance for Algorithm 1 is bounded V ξ k
[ λ k ] ≤ m 2 B 2 (1 − α k ) 2 .
This result shows that the correlation can be controlled by choosing α k closed enough to 1.
Witha reduced correlation, the wrong direction generated by MOGA could be corrected to a positivedirection, and thus the algorithm would be able to converge.
Note that the momentum on λ k onlyrequires an additional negligible computation cost.
On the other hand, the proposed exponential moving average mechanism also preserves the advantageof MOGA by carefully setting the momentum parameter α k to be 0 and asymptotically convergent to1 .
Intuitively, CRMOGA performs similarly to vanilla MOGA at the initial stage, and approximateslike fixed linear scalarization in the late stage.
Initially, the stochastic noise occupies a relativelysmall component of the stochastic gradient.
Thus, methods like MGDA, PCGrad, CAGrad can searchfor good common descent directions.
When the number of iterations increases and the solutionapproaches the Pareto optimal, the impact of the stochastic noise stands out, causing the inversedirection problem for MOGA as in the previous section.
At this time, α k becomes larger, maintainingrelatively stable λ k eliminates the strong correlation and guarantees the convergence.
Convergence analysis.
We next provide convergence analysis for CRMOGA 1 concluding allalgorithms based on the general framework 3.
Before that, we introduce basic assumptions.
Assumption 3.
f 1 ( x ) , . .
f m ( x ) are all differentialable, H -Lipschitz and L -smoothness, suggesting that for all x , y ∈ K , i ∈
[ m ] , it holds ∥∇ f i ( y )
∥ ≤ H and ∥∇ f i ( x ) − ∇ f i ( y )
∥
≤ L ∥ x − y ∥ .
4 Iterations × 100.
T r a i n i ng
Lo ss - L
MGDACR-MGDA
PCGradCR-PCGrad
CAGradCR-CAGrad (a) Training Loss-L    4 Iterations × 100.
T r a i n i ng
Lo ss - R
MGDACR-MGDA
PCGradCR-PCGrad
CAGradCR-CAGrad (b) Training Loss-R    4 Iterations × 100.
T e s t
i ng
Lo ss - L
MGDACR-MGDA
PCGradCR-PCGrad
CAGradCR-CAGrad (c) Testing Loss-L    4 Iterations × 100.
T e s t
i ng
Lo ss - R
MGDACR-MGDA
PCGradCR-PCGrad
CAGradCR-CAGrad (
[Figure]
Testing Loss-R
Lipschitz and smoothness are commonly used in complexity analysis for stochastic gradient algorithms [21, 15, 13, 4].
Based on Definition 3, we prove the following key results for CRMOGA.
Theorem 2 (Convex) .
Under Assumption 1, 2, 3.
For the sequence x 0 , x 1 , . . .
, x n generated by
Algorithm 1, we assume objective functions f 1 ( x ) , . . .
, f m ( x ) are all convex with bounded optimalvalues as f 1 ( x ∗ ) , . . .
, f m ( x ∗ ) ≤ F for x ∗ in Pareto set, and the distance from sequence to Pareto setis bounded, i.e., ∥ x k − x ∗ ∥ ≤ D .
Set 0 ≤ η n ≤ . . .
≤ η 1 ≤ 1 /mLB, α k ∈ (0 , 1] for k = 1 , . .
, nin Algorithm 1, it achieves 1 n
[Equation]
[Equation]
Theorem 3 (Non-convex) .
Under Assumption 1, 2, 3. Under Assumption 1, 2, 3.
For the sequencex 0 , x 1 , . . .
, x n generated by Algorithm 1, we assume objective functions
f 1 ( x k ) , . . .
, f m ( x k ) ≤ Fare all bounded for k = 1 , . . . , n .
Set 0 ≤ η n ≤ . . .
≤ η 1 ≤ 1 /mLB and α k ∈ (0 , 1] in Algorithm 1,it achieves
[Equation] [Equation]
Remark.
Note that our theoretical results imply that CRMOGA is convergent in averaging bychoosing suitable η and α .
Such an averaging analysis scheme is adopted by CAGrad [31], andis often used in stochastic optimization [42, 16, 4], e.g. Adagrad [29], Adam [25, 39].
It can betransformed into the traditional version, as shown in Appendix E.
The above results falls as an immediate corollary of the above Theorem.
Corollary 1.
Set 1 ≥ η k
= O (1 / √ k ) , α k = max { 1 −
η k /η 1 , 1 − η k /η 1 ( k (cid:80) mi =1 | λ ik − 1 − ˆ λ ik | ) } in
Theorem 2,3, Algorithm 1 converges with O ( n 1 / 2 ) , O ( n 1 / 4 ) rates in the convex and non-convex case.
This result indicates that Algorithm 1 has a convergence rate that matches the one in the singleobjective stochastic setting [7, 2].
It demonstrates the efficacy of the proposed momentum mechanism.
Remark.
One may argue that choosing α k ≈ 1 lets CRMOGA approximate linear scalarization thatis guaranteed to converge to Pareto optimal, and may impair the practical advantage for gradientalteration.
Note that the coefficient α k = 1 − 1 / √ k makes CRMOGA significantly different fromfixed linearization.
Such a parameter suggests the possible accumulated variation for λ k is about 328 (cid:80) nk
=1 1 / √ k
= O ( √ n ) , which preserves an enough space for effective gradient alteration.
Experiments
In this section, we present empirical results on both synthetic with convex objectives and real-worlddatasets for multi-task learning with deep neural networks, representing convex and nonconvexsettings, respectively.
We apply CRMOGA to MGDA, PCGrad, CAGrad, and use CR- to distinguishwhether it adopts our approach.
We conduct experiments to answer the following questions:
Q1:
Does our approach fix the non-convergence and improve the optimality for MOGA algorithms?
Q2:
Does our approach consistently improve the practical performance of MOGA in deep learning?
[Table]
Q3:
Is our approach flexible to be implemented and agnostic to models as well as optimizers?
Synthetic Experiments To demonstrate whether CRMOGA fixes the convergence issue for gradientalteration algorithm, we first consider the same simple convex setting as the example 4 in Section4.
We initialize x 0 = (0 , 0 . 4) and use SGD as the optimizer.
To enable fair comparison, theadaptive stepsize is set with η k = 0 .
006 / √ k for all the algorithms, and the momentum parameter forsmoothing the composite weights is set with the suggested parameter setting in Corollary 1.
Figure 1 illustrates the multi-objective losses and distance to the Pareto set for this problem.
We firstnote that both the two objective losses of MGDA, PCGrad, CAGrad are significantly larger than theircounterparts applied with the proposed mechanism.
Furthermore, the losses are even increasing atthe initial stage as a result of the inverse direction issue demonstrated in Figure 1a.
It moves x k awayfrom the Pareto set, which unfortunately causes an increasing suboptimal gap.
In contrast, applying
CRMOGA enhances the algorithms to converge to Pareto optimal.
This positively answers Q1 .
We repeat the means over 3 runs.
[Table]
Multi-Task Classification For multi-task classification scenarios, we follow the experiment setup from Sener et al.
[41],and consider the MultiMNIST dataset [40] with 60K examples.
MultiMNIST contains two tasks: classifying the digit on thetop-left (task-L) and classifying the digit on the bottom-right(task-R).
We use the LeNet architecture [26], and use SGDas the optimizer.
The stepsize and parameters are tuned fromvanilla MGDA, PCGrad, CAGrad, and the correlation-reducedcounterparts follow the same parameter setting to enable a faircomparison.
The momentum parameter is set with suggestedvalue in Corollary 1.
We report the train and test losses in Figure 2.
We can see that algorithms with λ momentum performbetter than original ones with respect to both train and test losses.
We also observed that λ momentumconsistently improves the accuracy for all the objectives in Table 2.
We hence positively answer Q2 .
Semantic Segmentation For semantic segmentation scenarios, we follow the experiment setup from
Yu et al.
[50] and Liu et al.
[31], and consider the CityScapes.
CityScapes similarly contains twotasks: 7-class semantic segmentation and depth estimation.
We combine algorithms with MTAN[33] based on SegNet architecture [1].
We use adam [25] as the optimizer.
All parameters are tunedfrom vanilla MOGA algorithms, and the correlation-reduced counterparts follow the same parametersetting to enable a fair comparison.
The momentum parameter is set with the suggested value.
We report mean Intersection-Over-Union (mIoU), pixel accuracy (Pix Acc), and absolute/relative(abs/rel) errors in Table 1.
We observe consistent improvement when combining our approach withthe three gradient alteration methods.
In particular, the proposed mechanism enhances the optimalityof solutions with respect to both train and test losses.
Combining the above results, we demonstratethat our methods work for different models and optimizers.
We hence give a positive answer to Q3 .
Conclusions
In this work, we study multi-objective gradient alteration in the stochastic setting and identify afundamental flaw in these algorithms.
We prove their non-convergence issue of them via an illustrativeexample.
We give fixes to this problem by slightly changing the algorithms.
We prove the convergenceof the resulting algorithms and show that they yield consistent empirical performance improvement.
Multi-Objective Gradient
Alteration
and Beyond”
We organize the supplemental materials as follows.
Section A provides a detailed comparisonbetween weak Pareto optimal and Pareto Optimal.
Section B provides the derivation of PCGradwith more than two objectives to our general framework.
Section C discusses more details of theconvergence analysis framework in Definition 3.
Section D supplements the theoretical results on thenon-convergence for MOGA algorithms.
Section E supplements the convergence results for bothconvex (Theorem 2) and non-convex
(Theorem 2) cases.
In particular, we discuss the averagingscheme in more detail, and provide a way to transform it to traditional convergence using a typicaluniformly sampling technique in stochastic convergence analysis [4, 27, 8, 23].
Note that we fix theresults of Theorem 2 and Theorem 3 in the main paper.
Correspondingly, the parameter setting of α kis slightly changed in Corollary 1, and does not affect the claim nor the empirical results.
Section G.3provides an additional simulation result with more general Gaussian noise.
Section G.4 introducesthe implementation details of experiments.
A Detailed Comparison between Weak Pareto Optimal and Pareto Optimal
In this section, we give a rigorous comparison between the definitions of Pareto optimal and weak
Pareto optimal, where the latter is defined as
Definition 4 (Weak Pareto optimal) .
A solution x ∗ ∈ K is called weak Pareto optimal if there is notx ∈ K such that F ( x ) <
F ( x ∗ ) , i.e. f i ( x ) ≤ f i ( x ∗ ) for all i = 1 , . . .
, m .
By the Definition 1 and Definition 4, it can be easy to know that Pareto optimal points are alwaysweak Pareto optimal, and the converse is not always true.
Take a two-objectives example, the minimalpoints for f 1 are all Pareto weak optimal, but only ones that are also optimal for f 2 are Pareto optimal.
Furthermore, if we assume f 1 has a unique minimizer, which is then Pareto optimal.
In rigorous words, we state below that relation among Pareto optimal and weak Pareto optimal, as arigorous version for Proposition 1 (b).
Proposition 3 (Theorem 5.13 and Lemma 5,14 in [18]) .
For MOO problem 1, the following statementshold: (a) For convex objectives F ( · ) .
If there exists λ ∈ S m such that x ∗ = arg min x ∈K λ T F ( x ) ,then x ∗ is weak Pareto optimal.
(b) Further if x ∗ is the unique minimizer, then x ∗ Pareto optimal.
For ease to understand, previous studies [12, 34, 31] do not distinguish the detailed difference betweenthe above definitions.
This paper also follows this presentation trick.
B More Details of the General Framework for MOGA
PCGrad with more than 2 objectives.
Specifically, the PCGrad is initialized by v PC i = ∇ f i ( x k ) .
When subtracting the conflicting component, i.e., v PC i = v PC i − v PC i ⊤ ∇ f j ( x k ) ∥∇ f j ( x k ) ∥ 2 ∇ f j ( x k ) , it is equivalentto subtracting a weight v PC i ⊤ ∇ f j ( x k ) ∥∇ f j ( x k ) ∥ 2 ∇ f j ( x k ) for gradient v j if using the framework 2.
Providedwith such equivalence, d k = − 1 m (cid:80) mi
=1 v PC i is equivalent to d k = −∇ F ( x k )
λ k using Algorithm
1: Input: ∇ f i ( x k ) , i = 1 , . . .
, m 2: Initialize: v i = ∇ f i ( x k ) , v PC i = v i , λ i = 1 for i = 1 , . . .
, m 3: for i ∈ [ m ] do 4: for j uniformly ∼ [ m ] \ i in random order do 5: if v PC i ⊤ ∇ f j ( x k ) < 0 then 6:
Set v PC i = v PC i − v PC i ⊤ ∇ f j ( x k ) ∥∇ f j ( x k ) ∥ 2 ∇ f j ( x k ) 7:
Update λ j = λ
j − v PC i ⊤ ∇ f j ( x k ) ∥∇ f j ( x k ) ∥8: end if 9: end for 10: end for 11: return λ
C More Details of the Convergence Analysis Framework
In this section, we provide more details about our stochastic convergence analysis framework.
Wefirst offer proof details of the Proportion 1, and then generalize previous analysis based on it.
Finally,we give a discussion for Definition 3 that is the stochastic convergence framework based on theprevious full-batch setting.
C.1 Proof of Proportion
Proof.
The Pareto criticality can be equivalently defined as: a x ∗ ∈ K is Pareto critical,if min λ ∗ ∈ S m ∥ (cid:80) mi =1 λ ∗ i ∇
f i ( x ∗ ) ∥
= 0 , where S m = { ( λ 1 , . .
λ m ) | (cid:80) mi =
λ i = 1 , λ ∈[0 , 1] } denotes the probability simplex.
Therefore, if there exist a λ ∈ S m , such that 0 ≤∥ (cid:80) mi
=
λ i ∇ f i ( x ∗ ) ∥
= 0 , then min λ ∗ ∈ S m ∥ (cid:80) mi =1 λ ∗ i ∇ f i ( x ∗ )
∥
≤ ∥ (cid:80) mi
=
λ i ∇ f i ( x ∗ ) = 0 .
Wethus prove Proposition (a) , and Proposition (b) directly comes from Proposition 3.
C.2 More Details of Previous Convergence Analysis in Full-batch Gradient Setting
Previous convergence analyses in full-batch gradient setting, including MGDA [12, 43], PCGrad [50],
CAGrad [31] can be analyzed based on Proposition 1 that reflects Pareto optimality and criticality,which induces the following convergence analysis framework.
Definition 5.
(a)
A
sequence { x k } ∞ k =1 asymptotically converges to Pareto critical points if
lim k →∞ min λ ∗ k ∈ S m ∥∇ F ( x k ) λ ∗ k ∥ → 0 .
(b) For convex multi-objective function F ( · ) .
A sequence { x k } ∞ k =1 asymptotically converges toPareto optimal if lim k →∞ max x ∗ k ∈K min λ ∗ k ∈ S m ( λ ∗ k ⊤ F ( x k ) − λ ∗ k ⊤ F ( x ∗ k ))
→ 0 .
a
) can be reviewed as the limit for attaining the criticality condition (the equivalent definition in[31]).
(b) is equivalent to that there exists a composite weight λ ∗ k ∈ S m such that λ ∗ k ⊤ F ( x k ) −min
x
∗
k
∈K
λ ∗ k
⊤
F x
∗
k ) → 0 , which is essentially the limit for attaining Pareto optimal condition in
Proposition 1.
It can be also reviewed the the convergence for multiobjective metric function [43].
We next briefly introduce the analysis of MGDA [12, 43] as an typical example that provides both nonconvex and convex convergence.
It proves that the norm of min-norm directionmin λ ∗ k ∈ S m ∥∇ F ( x k ) λ ∗ k
∥ is asymptotically convergent to zero with an order of O (1 / √ n ) , which isreviewed as the convergence rate with non-convex functions.
Theorem 4 (Theorem 3.1 in [12]) .
For MGDA algorithm updating with stepsize η ≤ 1 /L n n (cid:88) k
=
[Equation]
This result also implies that the composite gradient
d k = ∥∇ F ( x k ) λ ∗ k ∥ for MGDA asymptoticallyvanishes, which indicates that the sequence x k iterated by gradient descent using d k limits to acritical point x ∗ .
If the multi-objective function is convex for each objective, such a critical point x ∗ is also weak Paretooptimal [44].
It is also not difficult to verify that if x ∗ is a Pareto stationary point, then it satisfies∇ F ( x ∗ ) λ ∗ = 0 for one λ ∗ ∈ S m .
If the functions are convex, this meets the optimal condition of thecomposite function λ ∗⊤ F ( x ∗ ) .
And by Proposition 1, we then know that x is weak Pareto optimal.
It can prove that (cid:80) nk
=1 λ ∗ k ⊤ F ( x k ) − (cid:80)
nk
=1 λ ∗ k ⊤ F ( x ∗ ) ≤
O (1) .
Since in the batch-gradientsetting, d k is the common descent direction such that F ( x k ) is decreasing with respect to k .
Thusit has
[Equation]
Therefore, we have the following theorem
Theorem 5 (Theorem 4.1 in [12]) .
For MGDA algorithm updating with stepsize η ≤ 1 /L
[Equation]
Where the averaging weight λ ∗ k n is also in S m .
Since in the proof, the comparing point x ∗ can be generated to arbitrarily chosen u ∈ K [45].
We can further generate the result with (cid:16) (cid:80) nk =1 λ ∗ k n
[Equation] [Equation]
) ≤ O (1 /k ) .
By letting u = arg min x ∈K (cid:16) (cid:80) n k
[Equation]
1 λ
[Equation]
k n
[Equation]
⊤ F ( x ) , it finally has
[Equation]
Similarly, PCGrad [50] is proved to converge at a Pareto critical point (Theorem 1 in [50]).
CAGrad
[31] generalizes the analysis of MGDA, and shows that it converges to the Pareto critical pointthat tends to be the critical point of the average loss when c ∈ [0 , 1) .
These convergence results canall be analyzed under the Pareto convergence framework as Definition 5.
C.3 More Discussion for Definition
Definition 3.
(a) A sequence { x k } ∞ k =1 asymptotic converges to Pareto critical points iflim k →∞ E [min λ ∗ k ∈ S m ∥∇ F ( x k ) λ ∗ k ∥ ]
→ 0 .(b)
For convex multi-objective function F ( · ) .
A sequence { x k } ∞ k =1 asymptotic converges to Paretooptimal if lim k
[Equation]
] → 0 .
Definition 3 is a stochastic version for Definition 5.
Similarly, (a) can be reviewed as the expectedlimit for attaining the criticality condition.
(b) is equivalent to that there exists a composite weightλ ∗ k ∈ S m such that E [ λ ∗ k ⊤ F ( x k )
− min x ∗ k ∈K λ ∗ k ⊤ F ( x ∗ k )]
→ 0 , which means λ ∗ k ⊤ E [ F ( x k )]
→min x ∗ k ∈K λ ∗ k ⊤ F ( x ∗ k ) , suggesting that the expected function value E [ F ( x k )] are optimal.
Thus,
Definition 3 is indeed derived from Pareto optimality and criticality.
D Missing Proofs for the Non-convergence of MOGA
In this section, we give detailed proof for the non-convergence for MGDA, PCGrad, CAGrad withthe example introduced in Section 4.
We then prove the non-Lipschitz of the composite weight forthe above MOGA algorithms.
D.1 Proof of Theorem
Proof.
Recall the two-dimensional stochastic optimization setting over the domain K = { ( x, y ) |x ∈
[ − 1 , 1] , y ≥ 0 }
[Equation]
The expected function is f 1 ( x, y ) =
( x + 3)
+ y 2 + 8 , f 2 ( x, y ) =
( x − 3)
+ y 2 + 8 .
Therefore,the optimization goal is to minimize the distance towards ( − 3 , 0) and (3 , 0) simultaneously.
We caneasily know that the Pareto set is the line segment { ( a, 0) | a ∈ [ − 1 , 1] } .
We can calculate the stochastic gradient in the above setting as
[Equation]
Next, we prove that for all point in domain { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the expected compositedirections E
[ d k ] =
( d 1 k , d 2 k ) of MGDA, PCGrad, CAGrad have positive value in y-axis, i.e. d 2 k > 0 ,which forces the algorithms to move away from the Pareto set.
Lemma 3 (Sener et al.
The min-norm solver has closed form solution for two-objective case.
Specifically, for g 1 , g 2 , the composite weight λ for g 1 can be calculated as
[Equation]
where the operator [ a ] + = max { a, 0 } .
Lemma 4.
For any point in domain { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , MGDA has a expectedcomposite directions
E [ d k ] =
( d 1 k , d 2 k ) such that d 2 k > 0 .
Proof.
We adopt simple enumeration method to calculate the expected direction.
Case 1: when the received gradients are
[Equation]
From 5, we can get λ
=
(1 − a ) / 2 , then the direction for ( a, b ) is
[Equation] [Equation]
Case 2: when the received gradients are
[Equation] [Equation]
From 5, we can get λ
=
(5 − a ) / 10 , then the direction for ( a, b ) is
[Equation]
Case 3: when the received gradients are
[Equation]
From 5, we can get λ
= ( − 3 a + 2 b + 19) / 26 , then the direction for ( a, b ) is
[Equation]
Case 4: when the received gradients are
[Equation]
From 5, we can get λ
= ( − 3 a − 2 b + 7) / 26 , then the direction for ( a, b ) is
[Equation]
By summing up the above cases and multiply the probability, we have
E [ d k ] =
( − 4 13 a, 12 − 22 b) .
Since we know that b ≤ 0 .
05 , we then know d 2 k ≥ 12 − 22 ∗ 0 .
05= 10 . 9 13 > 0 .
We thus prove the
Lemma.
Lemma 5.
For the point in domain { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , PCGrad has a expectedcomposite directions
E [ d k ] =
( d 1 k , d 2 k ) such that d 2 k > 0 .
Proof.
Similar with the proof of MGDA, we also adopt simple enumeration method to calculate theexpected direction.
Case 1: when the received gradients are
[Equation]
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients has not conflicting,then we know the y-axis value of the composite gradient for ( a, b ) is 4 − 2 b .
Case 2: when the received gradients are
[Equation]
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients consistently conflict with each other, then we know the y-axis value of the composite gradient for ( a, b ) is
[Equation]
Case 3: when the received gradients are
[Equation]
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients consistently conflict with each other, then we know the y-axis value of the composite gradient for ( a, b ) is
[Equation]
Case 4: when the received gradients are
[Equation]
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients consistently conflict with each other, then we know the y-axis value of the composite gradient for ( a, b ) is
[Equation]
Sum up the above, it is easy to know d 2 k is a increasing function for b , and is creasing for a ∈ [ − 1 , 0]and decreasing for a ∈ [0 , 1] .
Plug in a = ± 1 , b = 0 .
05 , we can get d 2 k ≥ 0 . 2 > 0 .
Lemma 6.
For the point in domain { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , CAGrad with c ∈ [0 . 8 , 1] hasa expected composite directions
E
[ d k ] =
( d 1 k , d 2 k ) such that d 2 k > 0 .
Proof.
Similar with the proof of MGDA, we also adopt simple enumeration method to calculate theexpected direction.
Case 1: when the received gradients are
[Equation]
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients has not conflicting,then we know the y-axis value of the composite gradient for ( a, b ) is − 2 b +4+2 c
(cid:112) a 2 + ( b − 2) 2 ≥− 2 ∗ 0 .
+ 4 + 2 c (cid:112) 0 2 + (0 .
− 2) 2 = 3 .
8 c + 3 . 9 ≥ 6 . 94 .
Case 2: when the received gradients are
[Equation]
0.4 0.6 0.8 x 0.
y
Expected Direction (a) MGDA 1.0 0.8 0.6 0.4 0.2 0.0 0.2 0.4 0.6 0.8 x 0.
y
Expected Direction (b) PCGrad 1.0 0.8 0.6 0.4 0.2 0.
0.2 0.4 0.6 0.8 x 0.
y
Expected Direction (
[Figure]
We can observe that this biased direction can move algorithms away from the Pareto optimal set, which resists the algorithm to converge.
Additionally, the stationary points for all three algorithms does not lies near the Pareto set.
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients consistently conflictwith each other, then we know the y-axis value of the composite gradient for ( a, b ) is − 2 b − 4 −
[Equation]
Case 3: when the received gradients are
[Equation]
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients consistently conflictwith each other, then we know the y-axis value of the composite gradient for ( a, b ) is − 2 b +
[Equation]
Case 4: when the received gradients are
[Equation]
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients consistently conflictwith each other, then we know the y-axis value of the composite gradient for ( a, b ) is − 2 b +
[Equation]
Sum up the above, we can get d 2 k ≥ 2 . 5 > 0 .
Figure 3 shows the vector field for the composite direction calculated by MGDA, PCGrad, CAGrad,which aligns with the above Lemmas that indicate the expected direction is inverse to the Paretoset.
This problem implies that E [ x k +1 ] is more away from Pareto optimal than E [ x k ] if x k liesin domain { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } .
Thus the expected suboptimality gap is always > 0 ,which implies that MGDA, PCGrad, CAGrad do not converge to the optimal solution.
In addition, wecould also infer from the figure that, the three MOGA algorithms are stationary around (0 , 0 . 4 0 . 6) ,which represents a large expected suboptimality gap in this case.
D.2 Proof of Proposition
Proof.
The proposition can be proved by the following simple 2 objectives case, where we areprovided with g
= (1 , 0) , g 2 = (1 − ε, 0) , g ′ 2 =
(1 + ε, 0) where ε << 1 .
For this function,the min-norm direction is g 2 for the first case, while is g 1 for the second case, which meansλ ( g 1 , g 2 )
=
(0 , 1) and λ ( g 1 , g ′ 2 ) =
However, we know that g 2 , g ′ 2 only have 2 ε gap.
Whenε limits to 0, there exists no finite β to bound ∥ λ ( g 1 , g 2 )
− λ ( g 1 , g ′ 2 ) ∥ / ∥ g 2 − g ′ 2 ∥ .
Generalize to PCGrad.
This property can be generalized to PCGrad, we next provide anothercounterexample to prove it.
Proof.
Suppose we are provided with g 1 = (1 , 0) , g 2 = ( − ε, 0) , g ′
= ( ε, 0) , ε << 1 .
For thisfunction, g 1 conflicts with g 2 while aligned with g ′ 2 .
This makes λ ( g 1 , g 2 ) =
( 12 + ε 2 , 12 + 12 ε ) and λ ( g 1 , g ′ 2 ) =
However, we know that g 2 , g ′ 2 only have 2 ε gap.
When ε limits to 0, thereexists no finite β to bound ∥ λ ( g 1 , g 2 ) − λ
( g 1 , g ′ 2 )
∥ / ∥ g 2 − g ′
∥ = (cid:113) 1 + 1 ε 4 / 4 .
Generalize to CAGrad.
This property can also be generalized to CAGrad with parameter c ∈ (0 , 1 / 4)with the similar example with PCGrad.
When c ∈ [1 / 4 , + ∞ ) , CAGrad performs more like MGDA,it is not difficult to generalize the example of MGDA.
Proof.
The proposition can be proved by the following simple 2 objectives case, where we areprovided with g
= (1 , 0) , g 2 = ( − ε, 0) , g ′ 2 =
( ε, 0) , ε << 1 .
For this function, g 1 conflicts withg 2 while aligned with g ′ 2 .
This makes λ ( g 1 , g 2 ) =
( 1 − c (1 − ε ), 12 ) and λ ( g 1 , g ′ 2 ) =
( 1+ c (1+ ε ), 12 ) .
However, we know that g 2 , g ′ 2 only have 2 ε gap.
When ε limits to 0, there exists no finite β to bound∥ λ ( g 1 , g 2 ) − λ ( g 1 , g ′ 2 )
∥ / ∥ g 2 − g ′
∥ = c/ε .
E Missing Proofs for the Convergence of CRMOGA
We here give a rigorous convergence analysis of CRMOGA in the stochastic (mini-batch) gradientsetting.
We first fulfill the proofs for Lemma 1 and Lemma 2.
Before providing the final proof,we would like to introduce some useful and insightful lemmas that reflect the algorithmic propertyand can be used for the final proof.
Specifically, Appendix E.4 and E.5 decompose the convex andnon-convex convergence bounds into sub-terms bounded by lemmas from Appendix E.3.
E
.1 Proof of Lemma
Lemma 1.
Under Assumption 1, 2.
The correlation between λ k and G k can be bounded as∥ E [ G k λ k ] − E [ G k ] E [ λ k ] ∥ 22 ≤ mσ 2 V ξ k ( λ k ) , where V ξ k ( λ k ) =
E ξ k [ ∥ λ k − E ξ k
[ λ k ] ∥ 22 ] .
Proof.
Denote g i ( x k , ξ ik ) =
g i ( x k ) , i ∈ [ m ] for simplicity.
By the unbiasness of each stochasticgradient E [ g i ( x k )]
= ∇ f i ( x k ) for each objective, we have
[Equation]
The last equation is by the fact that
[Equation]
Further, by the fact that ∥ E [ a ] ∥ ≤ E [ ∥ a ∥ ] and the triangle inequality for the l 2 norm, we know that∥ E [ m
[Equation]
Where, since 2 ab ≤ βa 2 + 1 β b 2 for β > 0 , we can get
[Equation]
Plug in, we obtain
[Equation]
The last equation is by the definition of V ξ k .
Here we let β k
=
[Equation]
and apply all theabove result, we finally get
[Equation]
We thus prove the Lemma.
Remark.
Lemma 1 suggests that the biasness of the composite gradient is bounded by both thevariance of composite weights λ k and multiple stochastic gradients g i ( x k ) .
Therefore, there are twoeffective ways to decrease this bias: reduce the variance of λ k , or the variance of g i ( x k ) .
For thelatter way, one can adopt variance reduction framework [21, 4] for stochastic optimization.
However,it [21] requires global gradients that may be computational expensive, or slows down the convergencerate [4].
For the former one, the momentum mechanism on λ k is a simple and effective way, whichenhances both theoretical and empirical results without additional computational cost.
E.2 Proof of Lemma 2
Lemma 2.
Under Assumption 1, λ k variance for Algorithm 1 is bounded V ξ k
[ λ k ] ≤ m 2 B 2 (
[Equation]
α k ) 2 .
Proof.
By the definition of Algorithm 1, we know that
[Equation]
From the fact that E [ ∥ X − E
[ X ] ∥ 22 ] ≤ E [ ∥ X − a ∥ 22 ] for any a , we can get
[Equation]
The last inequality is by the fact 0 ≤ λ ik , ˆ λ ik ≤ B, i = 1 , . . .
, m .
E.3 Lemmas needed for Proving Thm2/Thm3
This subsection presents several useful Lemmas for proving Theorem 2 and Theorem 3.
Lemma 1measures the expected gap between − (cid:80) m i =
λ ik ∇ f i ( x k ) and d k .
The following Lemma bounds thedot production between them, which measures the local decrease of the composite functional value.
Lemma 7.
Under Assumption 1, 2, 3, for any gradient descent algorithm updated with compositegradient G k λ k , we have the following inequality
[Equation]
i = λ
ik ∇ f i ( x k ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)    .
Proof.
We first decompose the term into
E ξ k 
[Equation]
i
[Equation]

[Equation]

[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:88) i =
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
  .
The first term can potentially corrupt the effectiveness of optimization, and the second term measuresthe descent value.
We next bound the first term.
By definitions and decomposition, we get
[Equation]

[Equation]

[Equation]
i
[Equation]
i
[Equation]
A)
[Equation]

[Equation]
C) .
We then bound each term individually.
By Assumption 1 and Assumption 3, we know that λ
ik ∈[0 , B ] , ∥∇ f i ( x k ) ∥
≤ H, ∀ i = 1 , . . .
, m .
By Cauchy–Schwartz inequality, we further know that forthe term Aterm A = E ξ k
[Equation]
≤
[Equation]
(cid:34)(cid:32)
[Equation]
i =
BH (cid:33) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) m (cid:88) i =
( λ ik − E [ λ ik ])( ∇ f i ( x k ) − g i ( x k ))
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:35) ≤ mBH E ξ k (cid:34) m (cid:88) i =
| λ ik − E [ λ ik ] |∥ ( ∇ f i ( x k ) − g i ( x k )) ∥(cid:35) .
Similar with the proof in Lemma 1, by the fact that ab ≤ 12 β k a 2 + β k 2 b 2 for any β k > 0 , and by thelinearity of expectation, we can getterm A ≤ mBH 2 β k m (cid:88) i =
E
[Equation]
k (cid:2)
[Equation]
+
[Equation]
m (cid:88) i = E ξ k (cid:2) ∥∇ f i ( x k )
[Equation]
g i ( x k ) ∥ 22 (cid:3) ≤ mBH
[Equation]
+
[Equation]
m (cid:88) i =
V [ g i ( x k )] .
By setting β k = (cid:112) V ξ k
[ λ k ] / (cid:113) (cid:80) m i
=
1 V ξ k
[ g i ( x k )] , we haveterm A ≤ mBH (cid:118)(cid:117)(cid:117)(cid:116)
V [ λ k ] m (cid:88) i =
V [ g i ( x k )] .
With similar tricks
[Table]
we have for the term B
[Table]
[Equation]
The first inequality is by Cauchy–Schwarz inequality.
The second one is by the triangle inequality ofl 2 norm, we have ∥ (cid:80) mi =1 ( λ ik − E [ λ ik ]) ∇ f i ( x k ) ∥ 2 ≤ (cid:80) mi =1 ∥ ( λ ik − E [ λ ik ]) ∇ f i ( x k ) ∥
= (cid:80) mi =1 |
λ ik − 744 E [ λ ik ] |∥∇
f i ( x k ) ∥ 2 .
With Assumption 3, we know (cid:80) mi =1 | λ ik − E [ λ ik ] |∥∇
f
i ( x k ) ∥ 2 ≤ H (cid:80) mi =1 | λ ik −
E [ λ ik ] | .
The third one is from the fact that ab ≤ 12 β k a 2 + β k 2 b 2 .
The forth one is also from CauchySchwartz inequality.
The last one is from Assumption 1 that states E [ λ ik ] ≤ B .
Further by settingβ k = (cid:112) V ξ k
[ λ k ] /B
(cid:113)(cid:80) mi
=1 V ξ k
[ g i ( x k )] , we haveterm B ≤ ( m + 1) BH (cid:118)(cid:117)(cid:117)(cid:116)
V [ λ k ] m (cid:88) i =
V [ g i ( x k )] ≤ mBH (cid:118)(cid:117)(cid:117)(cid:116)
V [ λ k ] m (cid:88) i =
V [ g i ( x k )] .
For term C, by the fact that only g i ( x k ) has randomness, we getterm C =
E ξ k   (cid:32) m (cid:88) i =
E [ λ ik ] ∇ f i ( x k ) (cid:33) ⊤ (cid:32) m (cid:88) i =
E [ λ ik ]( ∇ f i ( x k ) − g i ( x k ))
(cid:33)  =(cid:32) m (cid:88) i =
E [ λ ik ] ∇ f i ( x k ) (cid:33) ⊤ (cid:32) m (cid:88) i =
E [ λ ik ] (cid:0) ∇ f i ( x k ) − E ξ k
[ g i ( x k )] (cid:1)(cid:33) = 0 .
By summing up the above results, we obtain
E ξ k   (cid:32) m (cid:88) i = λ
ik ∇ f i ( x k ) (cid:33) ⊤ (cid:32)
d k + m (cid:88) i = λ
ik ∇ f i ( x k ) (cid:33)  ≤ 2 mBH (cid:118)(cid:117)(cid:117)(cid:116)
V [ λ k ] m (cid:88) i =
V [ g i ( x k )] .
Plug in the first decomposition in the beginning, we finally get
E ξ k  [Equation] (cid:32) m (cid:88) i =
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation] (cid:88) i =
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)    .
Remark.
The result from Lemma 7 indicates that the local decrease for expected composite loss E [ λ k ] T F ( x ) is bounded by 2 mBH
[Equation]
(cid:104)(cid:13)(cid:13)(cid:80)
[Equation]
The first term consists of the variances for composite weights and multiple gradients, and the second term is the negative norm of the effective direction.
When the solution approximates the optimal,the second term would be smaller than the former term when the variances are large, then the valueof the composite loss will increase, leading to the non-convergence.
This result is aligned with thenon-convergence for MOGA algorithms.
We next give a second-order bias analysis of the composite gradient d k by the following lemma.
Lemma 8.
Under Assumption 1, 2, 3, for any gradient descent algorithm updated with compositegradient G k λ k , we have the following inequality
[Equation]
Proof.
We first decompose the expectation into
E
[Equation]
k
[Equation]
(cid:125) [Equation] B .
We next analyze the term A. By the triangle inequality of the l 2 norm, we haveterm A =
E ξ k 
[Equation]
 [Equation]
Further, by the fact that λ ik ∈ [0 , B ] , i = 1 , . . .
, m , we know thatterm
A ≤ E ξ k
[Equation]
≤
[Equation]
We then analyze the term B. Similar with the proof in Lemma 7, and by the fact that the minus signdoes not affect the inequality, we know that
[Equation]
Combining all the results, we obtain
E
[Equation]
Remark.
In the single-objective stochastic optimization, the second term is zero, leaving only thegradient variance in the bound.
While in the multi-objective stochastic optimization, the secondterm is non-zero and related to the number of objectives, the variance of composite weight, and thegradient variance, which makes it significantly different.
E.4 Proof for Theorem 2 (Convex Convergence)
Theorem 2.
Under Assumption 1, 2, 3.
For the sequence x 0 , x 1 , . . .
, x n generated by Algorithm 1,we assume objective functions
f 1 ( x ) , . . .
, f m ( x ) are all convex with bounded optimal values asf 1 ( x ∗ ) , . . .
, f m ( x ∗ ) ≤ F for x ∗ in Pareto set, and the distance from sequence to Pareto set isbounded, i.e., ∥ x k − x ∗ ∥ ≤ D .
Set 0 ≤ η n ≤ . . .
≤ η 1 ≤ 1 /mLB, α k ∈ (0 , 1] for k = 1 , . .
. , n in
Algorithm 1, it achieves
[Equation]
To prove Theorem 2, it is sufficient to prove the bound for n
[Equation]
We next provide the reason.
Specifically, from Assumption 1, we have (cid:80) mi =1 λ ik ≥ 1 for k = 1 , . . . , n .
Therefore, we know that
[Equation]
Furthermore, denote ¯ λ k = λ k / (cid:80) mj =1 λ jk .
By the property for the max-min operator, the left side ofthe above inequality can upper bound what we need to bound in Theorem 2 as
[Equation]
Therefore, the rest of the proof is to bound 6.
Before the final proof, we first introduce several
Lemmas.
The following Lemma give an induction as
Lemma 9.
Under the same assumption as Theorem 2, for any gradient descent algorithm updatedwith composite gradient G k λ k , and any x ∗ in Pareto set, we have the following inequality
E ξ k
[ m
[Equation]
i =
[Equation]
m
[Equation]
V [ g i
[Equation]
x
[Equation] [Equation]
Proof.
From the L -smoothness of each objective function, we have
[Equation]
By the iterative rule, we know that x k +
=
x k + η k d k .
Further, by the convexity of f i , and denotex ∗ as any point in the feasible set, we can get
[Equation]
Summing up above for i = 1 , . .
, m , we havem
[Equation]
By setting η k ≤ 1 /mLB , we can know that Lη k
[Equation]
mi =1 λ ik (cid:1) ≤ Lη k mB ≤ 1 .
Then, the third termcan be bounded as Lη 2 k 2 (cid:0)(cid:80) mi =1 λ ik
[Equation]
∥ d k ∥
[Equation]
≤
[Equation]
2 ∥
d k ∥ 22 .
Plugging such a bound for the third term, andtake the expectation on random variable ξ k , we then obtain
[Equation]
By the linearity of expectation, we rearrange the above term into
E ξ k
[Equation]
We next to bound the first term.
By rearranging the equation, we get
E ξ k 
[Equation]
=
[Equation]
k 
[Equation]
i
[Equation]
(cid:33)
[Equation]
( x k − x ∗ )   .
The last equation is by the fact that x k +
= x k + η k d k .
Since we know that
[Equation]
The second inequality is by the assumption that the set { x k , k = 1 , . .
, K } is a subset of a boundedset with diagram D , and the third one is from Lemma 1.
We now have bounded the term.
By plugging in, we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
i
[Equation]
(from Lemma 8) .
In addition, there is a slight different with the Lemma that we aim to prove: the left side is x k +1 notx k .
The following we fix this issue.
By the H-Lipschitz continuous for each objective, we know that
E ξ k
[Equation]
By the triangle inequality of norm and the definition for the composite gradient d k , we have
[Equation]
The third inequality is from the H-Lipschitz, and the last one is from Assumption 1.
By the fact thatab ≤ 12 β k a 2 + β k 2 b 2 for any β k > 0 , we can have
[Equation]
Let β k = B √ m/ (cid:112)(cid:80) mi
=1 V [ g i ( x k ] , we have E ξ k
[ ∥ d k ∥ ] ≤ B (cid:112) m (cid:80) mi =1 V [ g i ( x k ] .
Plug in theresult, we can get
[Equation]
Combining all the above, we prove the lemma.
By summing up the above lemma, we get the following one
Lemma 10.
Under the same assumption as Theorem 2, for any x ∗ in Pareto set, we have thefollowing inequality for CRMOGA algorithmn
[Equation]
Proof of Theorem 2.
From Lemma 2, we know that V ξ k
[ λ k ] ≤ m 2 B 2 (1 − α k ) 2 .
From Assumption 2,we know that V ξ k ( g i ( x k )) ≤ σ 2 .
Then plug into Lemma 9, we getm (cid:88)
[Equation]
Sum up the above inequality from k = 1 to k = n , we haven (cid:88)
[Equation]
i
[Equation]
Take expectation of ξ 1 , ξ 2 , . .
. , ξ n on the both sides, we can getn (cid:88)
[Equation]
By the fact that η k ≥ η k +1 , we then bound the first term asn (cid:88) k
=
[Equation]
Plug in the above result, we haven
[Equation]
We thus prove the lemma.
By observing Lemma 10 and the final version we need to proof, the only different is the comparedoptimum is x ∗ for all k not
the adaptive one x ∗ k = arg
min x ∈K (cid:80) mi =1 λ ik f i ( x ) , the following lemmameasures such a gap.
Lemma 11.
Under the same assumption as Theorem 2, for any x ∗ in Pareto set, we have thefollowing inequality for CRMOGA algorithm
E
[Equation]
min
[Equation]
m
[Equation]
i = λ
[Equation]
f
[Equation]
k ) (cid:35) ≤n (cid:88) l = l (1 − α k ) m (cid:88) i =
| λ il − ˆ λ il +1 |
Proof.
Denote ¯ λ i = (cid:80) nk
=1 λ ik /n .
Denote x ∗ =
arg min x ∗ ∈K (cid:80) nk
=1 (cid:80) mi =
λ ik f i ( x ∗ ) and x ∗ k =arg
min x ∗ k
∈K (cid:80) mi
=1 λ ik f i ( x ∗ k ) , k = 1 , . . . , n .
We first rearrange into
E (cid:34) n (cid:88) k
= m (cid:88) i = λ
ik f i ( x ∗ ) − n (cid:88) k
= m (cid:88) i = λ
ik f i ( x ∗ k ) (cid:35) =
E (cid:34) n (cid:88) k
= m (cid:88) i =
¯ λ i f i ( x ∗ ) − n (cid:88) k
= m (cid:88) i = λ
ik f i ( x ∗ k ) (cid:35) .
Since x ∗ is the minimum for ¯ λ ik f i ( x ) , we then have
E
[Equation]
By the assumption that objective functions f 1 ( x k ) , . . .
, f m ( x k ) ≤ F are all bounded for k =1 , . . .
, n , we get
E
[Equation]
As we know thatn
[Equation]
t =
( λ it − λ ik ) | = 1 n n (cid:88) k
= m (cid:88) i = | n (cid:88) t
= ( λ it − λ ik )
| ≤n m (cid:88) i = n (cid:88) k
= n (cid:88) t =
| λ it − λ ik | ≤ 1 n m (cid:88) i = n (cid:88) k
= n (cid:88) t
= t (cid:88) l
=
k | λ il − λ il +1 | =n m (cid:88) i = n (cid:88) k
= n (cid:88) t =
k t (cid:88) l
=
k | λ il − λ il +1 | = 2 n m (cid:88)
i = n (cid:88) k
= n (cid:88) l = k n (cid:88) t
= l | λ il − λ il +1 | =n m (cid:88) i = n (cid:88) k = n (cid:88) l = k ( n − l + 1) | λ il − λ il +1 | ≤m (cid:88)
i = n (cid:88) k
= n (cid:88) l = k | λ i l − λ i l +1 | =m (cid:88) i = n (cid:88)
l = l (cid:88) k
= | λ il − λ il +1 | =m (cid:88) i = n (cid:88)
l
= l | λ il − λ il +1 | =m (cid:88) i = n (cid:88)
l = l (1 − α k )
| λ il − ˆ λ il +1 | =n (cid:88)
[Equation]
The last two equation is by the definition of Algorithm
We are now ready to proof Theorem 2.
Combine Lemma 10 and Lemma 11, we haven (cid:88) k
=
[Equation]
By averaging the above, and the fact that η n ≤ η 1 , we prove the theorem.
E.5 Proof for Theorem 3 (Non-convex Convergence)
Theorem 3.
Under Assumption 1, 2, 3.
For the sequence x 0 , x 1 , . . .
, x n generated by Algorithm 1,we assume objective functions f 1 ( x k ) , . . .
, f m ( x k ) ≤ F are all bounded for k = 1 , . . . , n .
Set0 ≤ η n ≤ . . .
≤ η 1 ≤ 1 /mLB and α k ∈ (0 , 1] in Algorithm 1, it achieves
[Equation]
Similar to the convex case, to prove Theorem 3, it is sufficient to prove the bound for
[Equation]
where λ k is generated by MOGA or CRMOGA algorithms.
The reason is also similar to the convexcase, which is that 7 upper bounds the left side in Theorem 3 because
[Equation]
, where ¯ λ k = λ k / ( (cid:80) mi =
λ ik ) 2 ∈ S m .
We next present necessary lemmas.
Lemma 12.
Under the same assumption as Theorem 3, for any gradient descent algorithm updatedwith composite gradient G k λ k , we have the following inequality
[Equation]
Proof.
From the L -smoothness of each objective function, we have
[Equation]
Sum up both side for i = 1 , . . .
, m , and take the expectation on random variable ξ k , we can get
E ξ k [ m (cid:88) i =
[Equation]
The last inequality is by the Assumption that λ ik ≤ B .
From the result of Lemma 7, we can boundthe first term and obtain
E ξ k [ m (cid:88)
[Equation]
Then, adopting the result from Lemma 8, we know that
E ξ k [ m (cid:88)
[Equation]
g
[Equation]
By the fact that η k ≤ 1 /LmB , we further have
[Equation]
=
V ξ k ( g i ) .
By rearrangement, we therefore haveη k
[Equation]
Take expectation of ξ 1 , ξ 2 , . .
. , ξ n on the both sides, we finally get
[Equation]
k
[Equation]
Lemma 13.
For CRMOGA algorithm, and under the same assumption with Theorem 3, we haven (cid:88) k
=
[Equation]
Proof.
First, we can decompose the left side asn (cid:88)
[Equation]
By the definition of CRMOGA algorithm, we know that λ
ik = α k λ ik − 1 + (1 − α k )ˆ λ ik , i = 1 , . . .
, m ,where { ˆ λ ik } mi =1 are calculated by MOGA algorithms.
Then we have the following decomposition for the first term
[Equation]
Combining the above, and by the assumption that | f i ( x k ) | ≤ F, λ ik , ˆ λ ik ∈ [0 , B ] as well as thenonincreasing for η k , we therefore haven
[Equation]
x k
[Equation]
] ≤
[Equation]
=
[Equation]
Now, we can present the final proof.
From Lemma 2, we know that V ξ k
[ λ k ] ≤ m 2 B 2 (1 − α k ) 2 .
From Assumption 2, we know that V ξ k ( g i ( x k )) ≤ σ 2 .
Then plug into Lemma 12, we get
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) m (cid:88) i = λ
ik ∇ f i ( x k ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  
 ≤η k E [ m (cid:88) i = λ
[Equation]
The above is divided with η k / 2 for the both side.
Then, sum up the above inequality from k = 1 tok = n and take the result from Lemma 13, we haven
[Equation]
By averaging the above inequality
[Equation]
we prove the theorem
[Equation]
E.6 Proof of Corollary 1
Corollary 1.
(a) Set 1 /mLB ≥ η k
= O (1 / √ k ) , α k = max { 1 −
η k /η 1 , 1 − η k / ( η 1 k (cid:80) mi =1 | λ ik − 1 −ˆ
λ ik | ) } in Theorem 2, Algorithm 1 converges with O ( n 1 / 2 ) rates in the convex case.
(b) Set 1 /mLB ≥η k
= O (1 / √ k ) , α k = max { 1 −
η k /η 1 , 1 − η k / ( η 1 √ k (cid:80) mi =1 | λ ik − 1 − ˆ λ ik | ) } in Theorem 3, Algorithm
O ( n 1 / 4 ) rates in the non-convex case.
Proof.
For the convex case (a) , α k = max { 1 − η k /η 1 , 1 − η k / ( η 1 k (cid:80) i ∈ [ m ] | λ ik − 1 − ˆ λ ik | ) } means1 − α k = min { η k /η 1 , η k / ( η 1 k (cid:80) i ∈ [ m ] | λ ik − 1 − ˆ λ ik | ) } .
Plug in Theorem 2, we have
[Equation]
n
[Equation]
n
[Equation]
n
[Equation]
l
[Equation]
l (
[Equation]
For the non-convex case (b) , α k = max { 1 − η k /η 1 , 1 − η k / ( η 1 √ k max i ∈ [ m ] | λ ik − 1 − ˆ λ ik | ) } means1 − α k = min {
η k /η 1 , η k / ( η 1 √ k max i ∈ [ m ] | λ ik − 1 − ˆ λ ik | ) } .
Plug in Theorem 2, we have
[Equation]
Remark 1.
It is also not difficult to verify that setting α k = max { 1 − η k /η 1 , 1 −η k / ( η 1 k (cid:80) i ∈
[ m ] | λ i k − 1 − ˆ λ i k | ) } also lead to the same convergence in the non-convex case.
Forsimplicity, we present the unified version in the main text, and provide the complete version in the
Appendix.
Remark 2.
Since multi-objective optimization could reduce to single-objective optimization if themultiple objectives are the same, the convergence rate of stochastic MOO will be no better thansingle-objective SGD.
Note that our convergence rates align with those in single-objective SGD [7, 2],which suggests that our bounds are tight in order.
In comparison with stochastic MOGA, we havedemonstrated clearly in this paper that stochastic MOGA algorithms fail to converge to the Paretooptimal/critical points in Section 4, so comparing them is not very meaningful.
As for comparingwith full-batch MOGA, the difference between them is just like GD with SGD.
E.7 Discussion on Theorem 2 and Theorem 3
Average scheme.
The averaging scheme for Theorem 2 and Theorem 3 can be transformed totraditional ones.
It just needs to use a typical uniform sampling technique in stochastic optimization[4, 27, 8, 23] to modify the algorithmic output.
Specifically, the output is changed from x n to ¯ x n thatis uniformly sampled in { x 1 , x 2 , . . .
, x n } .
Then, by this randomness and the linearity of expectation, we can know that
[Equation]
and
[Equation]
Therefore, we recover the traditional convergence bound for ¯ x n .
Assumptions of bounded gradient.
The assumption of bounded gradients appears very widely inrelated literature on stochastic optimization.
Specifically, in the convex setting, the bounded gradientassumption is used to derive the convergence bound for various famous optimizers, such as Adam(Theorem 4.1 in [25]), Adagrad (Theorem 5 in [9]), AMSGrad (Theorem 4 in [39]), etc.
In thenon-convex setting, it helps to analyze the stochastic convergence of momentum (Theorem 1 & 2in [48], Assumption 3 in [4], Assumption 2 in [49], and Assumption 1(ii) in [24]).
Additionally, inpractice with deep learning, there are many techniques that are designed to avoid gradient explosion,such as weight regularization [38] and gradient clipping [51].
These techniques ensure the boundnessof the gradient in practice, which validates this assumption.
Furthermore, our theoretical resultsonly require that ∥∇ f i ( x k ) ∥
≤ H for i = 1 , . . .
, m and k = 1 , . .
. , n , i.e., the gradients along theoptimization path are bounded, which is also supported by our experiments.
Assumptions of bounded function.
The assumption of bounded functions has been used in recentstudies on non-convex stochastic optimization, such as analyzing SGD (Assumption 2 in [10]), andanalyzing SGD+momentum (Assumption in Section 3 of [28]).
Actually, this assumption can befurther relaxed in our analysis.
Specifically, in the convex setting, it can be relaxed to that the functionvalues of Pareto optimal solutions are bounded, i.e., | f i ( x ∗ ) | ≤ F, i = 1 , . . .
, m , if x ∗ is Paretooptimal.
The reason is that we only require this condition in Lemma 11.
The relaxed assumption issurely satisfied since the optimal solutions naturally have bounded function values.
In the non-convexsetting, note that in the related papers analyzing SGD (Remark 2 in the analysis of SGD [7]) as wellas MOGA (Theorem 3.1 in the analysis of MGDA [12]) with non-convex functions, it is commonlyassumed that the initial function values
f i ( x 1 ) , i = 1 , . . .
, m are bounded.
Essentially, our analysisonly requires that f i ( x k ) , i = 1 , . . .
, m are bounded for k = 1 , . . . , n , since we only use it in Lemma13.
This assumption is mild in practice as long as the function values of iterates along the practicaltraining path are bounded.
In fact, all our experiments indicate that the function values are welllocated in a bounded regime.
F Convergence Analysis for Strongly Convex Functions
Theorem 6.
Under Assumption 1, 2, 3.
For the sequence x 0 , x 1 , . . .
, x n generated by Algorithm 1, we assume objective functions f 1 ( x ) , . . .
f m ( x ) are all µ -strongly convex and boundedas f 1 ( x ) , . .
f m ( x ) ≤ F , and the distance from sequence to Pareto set is bounded, i.e.,∥ x k − x ∗ ∥ ≤ D for x ∗ in Pareto set.
Set 0 ≤ η n ≤ . . .
≤ η 1 ≤ 1 /mLB, α k ∈ (0 , 1] fork = 1 , . .
. , n in Algorithm 1, it achieves
[Equation]
Proof.
By the property of µ -strongly convex ((4.12) from [2]), we have
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:88) i =
[Equation]
) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
, where x ∗ k = arg min x (cid:80) mi =1 λ ik f i ( x ) .
Recall Lemma 12, we know that
[Equation]
k
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
=
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]

[Equation]
(cid:34)
[Equation]
(cid:88) i =
[Equation]
(cid:35)
[Equation]
(cid:118)(cid:117)(cid:117)(cid:116)
[Equation] [Equation]
(cid:88)
[Equation]
=
[Equation]
V
[Equation] [Equation]
(cid:88)
[Equation]
=
V
[Equation]
( g i ) .
By combining the above, we getη k µ E ξ k (cid:34) m (cid:88) i =
[Equation]
f
[Equation]
− m (cid:88)
[Equation]
=
[Equation]
) (cid:35)
[Equation]
(cid:34)
[Equation]
By rearranging the inequality, we further have
E
[Equation]
By the optimality of x ∗ k , we know that (cid:80) mi =1 λ ik f i ( x ∗ k ) ≤ (cid:80) mi =1 λ ik f i ( x ∗ k +1 ) .
Therefore, we can get
E ξ k (cid:34) m
[Equation]
i =
[Equation] [Equation]
=
[Equation]
+
[Equation]
Further by decomposing, we can obtainm (cid:88) i =
[Equation]
=
[Equation]
i =
[Equation]
ik
[Equation]
x ∗
[Equation]
+
[Equation]
By the definition of Algorithm 1 and the boundness of gradient function.
We know that λ
ik +1 − λ ik =(1 − α k +1 )
(ˆ λ ik +1 − λ ik ) and | f i ( x k ) | ≤ F, | f i ( x ∗ k ) | ≤ F.
Then, we can easily get that − ( λ ik −λ ik +1 )
f i ( x k +1 ) ≤ F (1 − α k +1 ) | ˆ λ ik +1 − λ ik | and ( λ ik − λ ik +1 ) f i ( x ∗ k +1 ) ≤ F (1 − α k +1 ) | ˆ λ ik +1 − λ ik | .
Thus, we havem (cid:88)
[Equation]
∗ k
[Equation] [Equation]
Then combining the above, we now obtain
[Equation]
From Lemma 2, we know that V ξ k
[ λ k ] ≤ m 2 B 2 (1 − α k ) 2 .
From Assumption 2, we know that
V ξ k ( g i ( x k )) ≤ mσ 2 .
Then plug into the above, we get
E ξ k
[Equation]
=
[Equation]
ˆ λ ik +1 − λ ik | .
Taking total expectations, this yields
E
[Equation]
m (cid:88)
[Equation]
We now are able to provide the convergence analysis for Algorithm 1 with fixed stepsizes.
Corollary 2.
Set 0 < η k = η ≤ min { 1 mLB , 1 L } and α k = max { 1 − η, 1 − η 2 / ( (cid:80) mi =1 | λ ik − 1 − ˆ λ ik | ) }in Theorem 6.
Algorithm 1 linearly converges to a solution near the Pareto optimal, and the expectedgap limits to ηM/µ , where constant M = 4 m 5 / 2 B 2 H
+ Lm 3 B 2 σ 2+ 2 F .
Proof.
From the parameter setting, we know that 1 − α k = min { η, η 2 / ( (cid:80) mi =1 | λ ik − 1 − ˆ λ ik | ) } , whichsuggests that 1 − α k ≤ η and (1 − α k +1 ) (cid:80) mi =1 | ˆ λ ik +1 − λ ik | ≤ η 2 .
By Theorem 6, we have
E (cid:34) m (cid:88) i = λ
ik +1 f i ( x k +1 ) − m (cid:88) i = λ
ik +
f i ( x ∗ k +1 )
(cid:35) ≤ (1 − ηµ ) E (cid:34) m (cid:88) i = λ
ik f i ( x k ) − m (cid:88) i = λ
ik f i ( x ∗ k ) (cid:35) + 4 m 5 / 2 B 2 Hη (1 − α k ) + Lm 3 B 3 σ 2 η 2+ 2 F (1 − α k +1 ) m (cid:88) i = | ˆ λ ik +1 − λ ik | ≤ (1 − ηµ ) E (cid:34) m (cid:88) i
= λ
i k f i ( x k ) − m (cid:88)
i = λ i
k f i ( x ∗ k ) (cid:35) + η 2 (4 m 5 / 2 B 2 H
+ Lm 3 B 2 σ + 2 F ) =
(1 − ηµ ) E (cid:34) m (cid:88) i = λ
ik f i ( x k ) − m (cid:88) i = λ
ik f i ( x ∗ k ) (cid:35) + η 2 M, where M = 4 m 5 / 2 B 2 H
+ Lm 3 B 2 σ 2+ 2 F .
Then, subtracting the constant ηM/µ from both sides,we obtain
[Equation]
Observe that the above is a contraction inequality, by the fact that
[Equation]
The second inequality is by the range of η .
The result thus follows by applying the above contractioninequality repeatedly through iteration k = 1 , . .
. , n .
[Equation]
If n → ∞ , then the suboptimal gap for the solution will limit to ηMµ .
Remark.
Here, the algorithm does not converge to Pareto optimal but is a suboptimal solution near anoptimal solution.
This result aligns with the one in single-objective stochastic optimization, and thusis tight in order.
Note that this gap is unavoidable, as is the case in single-objective SGD, because thestepsize needs to tradeoff the stochastic variance and the convergence rate (the discussion of Theorem4.6 in [2]).
Additionally, the suboptimal gap is related to the scale of the stepsize η , and hence we canset sufficiently small η to get better convergence.
To guarantee the convergence to Pareto optimal solutions, we can use a diminishing stepsize as thecase in single-objective stochastic optimization [2].
The result is as follows.
Corollary 3.
Set η k = βk + γ , β > 1 µ where γ > 0 such that η k ≤ 1 mLB , and α k = max { 1 −η k /η 1 , 1 − η 2 k / ( η 1 (cid:80) mi =1 | λ ik − 1 − ˆ λ ik | ) } in Theorem 6.
Algorithm 1 linearly converges to Paretooptimal with O (1 /n ) rates.
Proof.
Specifically, we prove the following result for k = 1 , . . . , n
[Equation]
where v satisfies that v = max { β 2 M βµ − 1 , (1 + γ )( (cid:80) mi
=
λ
i 1 f i ( x 1 ) − (cid:80) mi =
λ
i 1 f i ( x ∗ 1 )) } .
From Theorem 6, we know that
E
[Equation]
where the constant in the above is defined as M = 4 m 5 / 2 B 2 H η+ Lm 3 B 2 σ 2+ 4 F η 1 for simplicity.
Thelast inequality is by the fact that η k ≥ η k +1 .
We then prove the corollary by mathematical induction.
First, the parameter setting here ensures the result holds for k = 1 .
Next, we assume the inequality ofthe theorem holds for k , which holds
E
[Equation]
Then by the parameter setting, for the case of k + 1 , we have
[Equation]
2 v, where the last inequality is by the condition of v .
By the fact that ( a − 1)( a + 1) ≤ a 2 for a > 0 ,we have k + γ − 1 ( k + γ ) 2 ≤ k + γ − 1 ( k + γ − 1)( k +1+ γ )
=
1 k +1+ γ .
Plug in the above, we can prove the result for k + 1 .
Thus, we theorem holds for k = 1 , . . . , n , which suggests an O (1 /n ) convergence.
Remark 1.
The result from Corollary 3 shows the convergence rate is aligned with single-objective
SGD with adaptive stepsize, suggesting that Algorithm 1 is tight in order with strongly convexfunctions.
Remark 2.
The bounded function assumption in Theorem 6 can be relaxed to the combination ofthe assumptions in the convex and non-convex settings.
Specifically, the function values of Paretooptimal solutions are bounded, i.e., | f i ( x ∗ ) | ≤ F, i = 1 , . . .
, m , and the function values of iteratesalong the practical training path are bounded, i.e., f 1 ( x k ) , . .
f m ( x k ) ≤ F for k = 1 , . .
n.
Iterations × 100.
l a m bda b i a s
MGDA CR-MGDA
PCGrad CR-PCGrad
CAGrad CR-CAGrad
G Additional Experiments and More Details
G.1 More Insight of CR-MOGA
[Table]
How the non-convergence reflecte in practice.
In fact, the similar situation reflected by the counterexample is not rare in practice, especially when the stochastic noises are large.
For example, wehave added an additional experiment of MultiMNIST by setting batch size from 8 to 2, representingincreasing levels of stochastic noises (a smaller batch size introduces a larger stochastic noise in themini-batch gradient).
We report the training losses for the final outputs of MGDA and CR-MGDAas follows, representing the comparison of MOGA and CR-MOGA.
From results of Table 3, weobserve that the loss of vanilla MGDA becomes significantly larger when the batch size is smaller,while CR-MGDA appears not sensitive to the batch size.
This possible reason is that the convergenceissue of MOGA becomes more severe with larger stochastic noise, which severely degrades theperformance of MOGA.
Since CR-MOGA is proven to converge to the Pareto optimal/critical, itis much more robust to stochastic noises.
Thus, in this regard, our method is more practical forreal-world applications.
Comparison with the full-batch MOGA.
Although there are no guarantees for the closenessbetween λ calculated by CR-MOGA and the actual λ calculated by MOGA with full-batch gradients.
As shown in Figure 4, we empirically observe that the λ k of CR-MOGA is very close to the actualλ ∗ k yielded by the MOGA with full batch gradients.
Specifically, the gaps between λ k of CR-MOGAand actual λ ∗ k are below 0.01 with low error bars in the toy example, while the gaps of vanilla MOGAalgorithms are all averagely larger than 0.10 with large error bars.
[Figure]
G.2 More Details for the Deep Learning Experiments
Error bars.
Due to the space limit, we do not report the error bars of the experiments for the
MultiMNIST and CityScapes datasets.
For MultiMNIST, we here provide the error bars for theaccuracy in Table 4, and the error bars for the curves of loss values in Figure 5.
For the CityScapes,we provide the error bars for all the metrics in Table 5.
[Table]
Time as x − axis.
Our modification is only smoothing the composite weight λ k , whose dimensionm is the same as the number of objectives, which is usually not very large (often 2 or 3).
Thus, it
[Table]
[Figure]
[Table]
[Figure]
introduces nearly negligible extra computational overhead.
Figure 6 shows the curve when x − axis isthe running time.
They look nearly identical to the original Figures 2, respectively.
G.3 Additional Stochastic Convex Simulation
In this simulation
[Figure]
we establish a similar setting with Section 4, where we aim to optimize x ∈ R 2to attain smaller distances to two fixed points
a = [ − 3 , 0] , b = [3 , 0] .
The first objective isf 1 ( x ) =
∥ x − a ∥
[Figure]
and the second one is f 2 ( x ) =
∥ x − b ∥ 22 .
The stochastic gradient for eachobjective can be represented as g 1 ( x ) = x − a + ξ 1 and g 2 ( x ) = x − b + ξ 2 .
The stochastic noisesξ 1 , ξ 2 are two dimensional random noise with zero-mean Gaussian distribution, where the covariancematrix for ξ 1 is [[3 , 2] , [2 , 3]] and for ξ 2 is [[3 , − 2] , [ − 2 , 3]] .
Iterations × 100.
D i s t an c e t o
P a r e t o
S e t
MGDA CR-MGDA
PCGrad CR-PCGrad
CAGrad CR-CAGrad
Figure 7 demonstrates the distance to Pareto set.
We observe that all three MOGA algorithms can notconverge to the Pareto optimal points, while all CRMOGA algorithms can approach the Pareto set.
This result is aligned with the result in Section 4, and the more general Gaussian noise shows that thenon-convergence is not special in stochastic multiobjective optimization.
[Figure]
G.4 Implementation Details of Experiments
[Figure]
The implementation for CRMOGA is very simple, just by adding one additional line that exponentiallysmoothes the composite weights to MOGA algorithms.
Note that this mechanism does not requireadditional computational cost and is flexible to be implemented in MOGA algorithms.
[Figure]
H Limitation and Future Works
[Figure]
This paper only discusses the typical MOGA algorithms with strict convergence guarantees, i.e.,
MGDA, PCGrad, and CAGrad.
Other empirical-driven methods without rigorous theoretical guarantees, such as RotoGrad [19], are too expensive to be included in our analysis.
We believe that theconvergence issue in that line of literature will be a very good future work to explore in depth.
[Figure]