On the Convergence of Stochastic Multi-Objective Gradient Manipulation and Beyond
Abstract
The conflicting gradients problem is one of the major bottlenecks for the effective training of machine learning models that deal with multiple objectives.
To resolve this problem, various gradient manipulation techniques, such as PCGrad, MGDA, and CAGrad, are developed, which directly manipulate the conflicting gradients to refined ones with alleviated or even no conflicts.
However, the existing design and analysis of these techniques have been mainly conducted under the exact full-batch gradient setting, ignoring the fact that they are primarily applied with stochastic mini-batch gradients.
In this paper, we point out that the stochastic gradient manipulation algorithms may fail to converge to Pareto optimal solutions.
Firstly, we show that these seemingly different algorithms can be summarized into a unified algorithmic framework, where the descent direction is given by the composition of the gradients w.r.t.
the multiple objectives.
Then we provide a simple two-objective convex optimization instance to illustrate the non-convergence issue in detail under the unified framework, where the directions yielded by PCGrad, MGDA, and CAGrad even reversely optimize all the objectives in expectation simultaneously. The analysis also shows that the non-convergence results from the determination of the composite weights solely by the instantaneous stochastic gradients.
To fix the non-convergence issue, we propose a novel composite weights determination scheme that exponentially averages the past calculated weights.
Finally, we show the resulting new variants of stochastic gradient manipulation converge to Pareto optimality and give rise to comparable or improved empirical performance.
Introduction
Multi-objective optimization (MOO) refers to the optimization of multiple objectives simultaneously [10, 6, 5].
MOO problems appear in many real-world scenarios, e.g., the online advertising model [35, 36, 29, 45] aiming to maximize both the Click-Through Rate (CTR) and the Post-Click
Conversion Rate (CVR); autonomous driving system [16, 34] learning to solve multiple tasks such as self-localization and object identification at the same time.
The main challenge lies in the inherent conflicts between multiple objectives that would impair the training [21, 48, 30].
For example, in online advertising, merely optimizing CTR or CVR will degrade the performance of the other [35, 36].
Various MOO algorithms [40, 48, 30, 44, 31, 18] were proposed to deal with the conflicting problem by altering the gradients evaluated from multi-objective losses.
Typically, multi-objective gradient manipulation (MOGM) methods such as MGDA [40], PCGrad [48], and CAGrad [30] adjust the multiple gradients to seek a common descent direction that simultaneously descents all the objectives.
While these algorithms have been successfully applied in a wide range of applications and achieved significant performance gains, the current design and analysis are only derived with exact (i.e., full-batch) objective gradients [40, 48, 30, 44, 31, 18].
However, in most real-world scenarios, evaluating stochastic (i.e., mini-batch) gradients is a much more practical and efficient method from the computational cost aspect, especially for training deep neural networks (DNNs) [24, 38].
In the single-objective case, since the stochastic gradients are unbiased estimators of the true gradients, directly using the stochastic gradient still maintains the right expected direction to descend the objective.
However, in the multi-objective case, since altering the stochastic gradients could amplify the negative impact of the stochastic noise, the common descent direction for the stochastic objectives may not be the right direction to descend the true objectives in expectation.
Such gap between full and stochastic gradients could cause non-convergence issues for directly using MOGM algorithms.
In this paper, we rigorously show that the above non-convergence issues indeed exist.
Surprisingly, we prove that altering the stochastic gradients yields a direction that even inversely optimizes all the objectives in expectation for some cases.
Such an inverse direction that moves the algorithms in the wrong direction will resist the solution from convergence.
This implies that current MOGM algorithms are suboptimal under stochastic gradient settings, which potentially degrades the performance of MOGM algorithms applied for training DNNs.
In particular, we make the following key contributions.
To uniformly analyze different MOGM algorithms, we summarize them into a general framework where the altered direction is a weighted composition of the multiple gradients, and the composite weights are functions of the multiple gradients determined by the alteration rules of different MOGM algorithms.
We provide strictly equivalent transformation to show that typical methods such as
MGDA, PCGrad, and CAGrad are all included in the summarized form.
This framework is general and incorporates a wide range of algorithms, such as recently proposed GradVac [44], IMTL-G [31].
Under the above general framework, we elucidate how the MOGM algorithms can cause nonconvergence by providing an example of a simple stochastic convex optimization problem, where
MGDA, PCGrad, and CAGrad provably do not converge to an optimal solution.
The key reason is that the composite weights are calculated only from the instantaneous gradients in one step, and thus inherit the same randomness as the stochastic gradients.
This strong correlation between the composite weights and stochastic gradients could lead to significant estimation bias of their composition, which forces the composite gradient in the wrong direction that even degrades all the objectives.
Since MGDA, PCGrad, and CAGrad possess typical properties among MOGM algorithms, our analysis easily extends to other MOGM algorithms contained in our general framework as well.
Furthermore, we point out that the previous analysis of the convergence for the MGDA algorithm [33] is established on an unjustified assumption, where the composite weights are Lipschitz functions w.r.t.
each gradient.
We rigorously prove that this assumption is not true for MOGM algorithms.
The above analysis suggests that in order to achieve guaranteed convergence, the optimization algorithms must reduce the correlation between the composite weights and stochastic gradients.
To resolve this issue while preserving the practical advantages of MOGM, we fix MOGM algorithms with a principled and practical momentum mechanism that exponentially averages the composite weights instead of just using the values computed from the instantaneous gradients.
Theoretically, momentum reduces the variance of the composite weight [4], then decreasing the correlation as well.
Thus, the inverse direction problem can be alleviated, and we prove that gradient manipulation algorithms can converge to optimal with the same rate as single objective stochastic optimization by adopting this mechanism.
Our approach is simple to implement and agnostic to model and optimizer, and hence can be easily plugged into various gradient manipulation algorithms.
Finally, we empirically verify our proposal in simulation and deep multi-task learning tasks, where we observe comparable or better performance.
Our advantage becomes more significant when the batch size is smaller, implying that the convergence issue is more severe with larger stochastic noise.
Preliminaries
In this section, we briefly review the basic background knowledge of multiple-objective optimization, and introduce typical algorithms as well as their convergence analysis.
2.1 Basic Concepts of Multiple-Objective Optimization
Multiple-objective optimization (MOO) is concerned with solving the problems of optimizing multiple objective functions simultaneously [50, 40], termed as
[Equation]
where m ≥ 2 denotes the number of objectives, and K ⊂ R d denotes the feasible set in a d dimension space.
Each objective f i : K
→ R , i ∈ { 1 , . .
. , m } is the i -th loss function.
Different from singleobjective optimization where solutions x , x ′ can be order by f ( x ) ≤ f ( x ′ ) or f ( x ) ≥ f ( x ′ ) .
In
MOO, it could have two parameter vectors where one performs better for task i and the other performs better for task j ̸
= i .
Therefore, Pareto optimality is defined to deal with such an incomparable case.
Definition 1 ( Pareto optimality ) .
For any two solutions x , x ′ ∈ K , we say that x dominates x ′ , denoted as x ≺ x ′ or x ′ ≻
x , if f i ( x ) ≤ f i ( x ′ ) for all i , and there exists one i such that f i ( x ) < f i ( x ′ ) ; otherwise, we say that x does not dominate x ′ , denoted as x ⊀ x ′ or x ′ ⊁ x .
A solution x ∗ ∈ K is called Pareto optimal if it is not dominated by any other solution in K .
Note that there is a set of Pareto optimal solutions, termed as Pareto set .
The goal of MOO is to find a Pareto optimal solution, which is necessary to be Pareto critical [3].
Definition 2 ( Pareto criticality ) .
A solution x ∗ ∈ K is called Pareto critical if there is no common descent direction d ∈ R d such that ∇ f i ( x ∗ )
⊤ d < 0 , i = 1 , . . .
m for all objectives.
This definition indicates that if x is not Pareto critical, such direction d will be a local descent direction for F at point x .
Optimizing through d in the local neighborhood of x is able to get a better solution that dominates x [10].
Since Pareto criticality reflects the local property compared with Pareto optimality, it is often used as the local minimal condition for MOO with non-convex objectives [11].
We then present sufficient conditions for determining Pareto criticality/optimality, which appear as metrics to study the convergence for MOO algorithm
Denotes the probability simplex as S m = { ( λ 1 , . . .
, λ m ) | (cid:80) mi
=
λ i = 1 , λ ∈ [0 , 1] } , we have the following proposition.
Proposition 1.
For MOO problem 1, (a) If there exists λ ∈ S m such that ∥ (cid:80) mi =
λ i ∇ f i ( x ∗ ) ∥
= 0 , then x ∗ ∈ K is Pareto critical.
(b) For convex objectives
f i ( · ) , i = 1 , . . .
, m .
If there exists λ ∈ S m such that x ∗ = arg min x ∈K λ T F ( x ) , then x ∗ is Pareto optimal 3 .
(a) is derived from the equivalent definition of Pareto critical points in [30], it reflects the stationary condition for MOO algorithms.
(b) refers to Theorem 5.13 and Lemma 5.14 in [17], it implies that the minimizer of any linearization is Pareto optimal for the multiple objectives.
2.2 Multi-Objective Gradient Manipulation
Similar to single-objective optimization, MOO can be solved by running iteratively with gradientbased algorithms.
Suppose n denotes the number of iterations.
At each k = 1 , . . . , n round, the key challenge is the gradient conflicting problem, i.e., ∇ f i ( x k ) ⊤ ∇ f j ( x k ) < 0 , i ̸
=
j, i, j ∈ [ m ]= { 1 , . . .
, m } ,
which leads to performance drop for an objective when optimizing another one [48], i.e. optimizing f i along the negative gradient direction −∇ f i ( x k ) will inversely optimize f j . multi-objective gradient manipulation (MOGM) algorithms aim to search for a direction
d k that is not conflicting with each negative gradient, i.e., −∇ f i ( x k ) d k > 0 , i ∈ [ m ] .
Using such a non-conflicting direction d k to execute the gradient descent step to update the decision i.e., x k +
= x k + η d k where η is the step size, is shown to get better performance in practice [40, 48, 30].
Mathematically, we can measure the decrease of each objective i ∈
[ m ]
[Equation]
by the first-order Taylor approximation assuming η is small [30].
Therefore, if − d ⊤ k ∇ f i ( x k ) > 0 for each objective i , then this direction is able to descend all objectives simultaneously.
There are various ways to compute such kind of direction, and we’ll introduce typical MOGM methods as follows.
Multiple Gradient Descent Algorithm (MGDA) [40] directly optimizes towards the Pareto criticality by leveraging Definition 2.
Specifically, in each iteration k , MGDA aims to find a direction d k to maximize the minimum decrease across the losses by solving the following subproblem
[Equation]
By regularizing the norm of d k on the right side, it computes the direction by
[Equation]
This sub-problem can be rewritten equivalently as the following differentiable quadratic optimization
[Equation]
If µ < 0 , then ∇ f i ( x k ) ⊤ d k < 0 , which means x k is not Pareto critical from Definition 2, and d k is the direction to descent all the objectives simultaneously [10, 11].
To simplify the optimization, such primal problem has a dual objective as a min-norm oracle
[Equation]
The direction is then calculated by d k = − {+(cid:80) mi =
+}
λ ik ∇ f i ( x k ) .
Projecting Conflicting Gradients (PCGrad) [48] alters the conflicting gradients by projecting each onto the normal plane of the other, preventing the interfering components of the gradient from being applied to the network.
Specifically, it initializes the projected gradients as v PC i = ∇ f i ( x k ) .
If projected gradient v i and raw gradient ∇ f j ( x k )
are conflicting, i.e., v ⊤ i ∇ f j ( x k ) < 0 , then it iteratively updates the projected gradients by
[Equation]
for i, j = 1 , . . .
, m until the projected gradients do not conflict with the received gradients.
Since the conflicting components are overall eliminated, it makes the direction d
k
= − 1 m (cid:80) mi =1 v PC i satisfies d ⊤ k ∇ f i ( x k ) < 0 for each objective i , thus optimizing all objectives simultaneously.
Conflict-Averse Gradient (CAGrad) [30] generalizes averaging linearization and MGDA.
Specifically, it constrains search region for the common direction as a circle around the average gradient v 0 = (cid:80) mi =1 1 m ∇ f i ( x k ) with diagram c ∥ v 0 ∥ .
Specifically, the direction d k is yielded by
[Equation]
It can minimize the average loss function, while leveraging the worst local improvement of individual tasks to regularize the algorithm trajectory by using the common descent property of MGDA.
Note that the gradient conflicting issue is not absolutely eliminated due to the new constraint.
Still, the above objective forces direction to approach the common descent direction as close as possible.
To simplify the optimization, the dual objective is
[Equation]
Then the direction is calculated by d k = − (cid:0) v 0 + c ∥ v 0 ∥ (cid:0)(cid:80) mi =1 λ ik ∇ f i ( x k ) (cid:1)(cid:1) .
Convergence analysis.
MGDA has been shown to converge to an arbitrary Pareto critical/optimal point with the same rate as single-objective optimization [11].
A similar result has been proved with PCGrad [48].
CAGrad has been shown to converge to the minimizer or stationary point of the averaging loss 1 m (cid:80) mi =
f i ( x ) when c ∈ [0 , 1) , or an arbitrary Pareto critical/optimal point when c ≥ 1 [30].
By Proposition 1, we can infer that the minimizer or stationary point of the averaging loss is also Pareto optimal or critical.
Hence, we’ll analyze them uniformly under the Pareto optimality in the following analysis, rather than distinguish them.
General Framework for Multi-Objective Gradient Manipulation
In this section, we summarize multi-objective gradient manipulation (MOGM) algorithms into a general framework with the goal of uniformly analyzing the above algorithms.
3.1 The Framework with Full-batch Gradients
Without loss of generality, we summarize the direction search of MOGM in the following framework.
[Equation]
where the composite weights λ k
= ( λ 1 k , . . .
, λ mk ) ⊤ is a function of the multiple gradients ∇ F ( x k ) =
( ∇ f 1 ( x k ) , . .
. , ∇ f m ( x k ))
⊤ .
Specifically, MGDA and CAGrad obtain λ k by their dual form of the subproblem that depends on the received gradients ∇ F ( x k ) .
For PCGrad, we take the twoobjective case as an example.
If the gradients are not conflicting, we have λ k
=
(1 / 2 , 1 / 2) as d k = − 12 ( ∇ f 1 ( x k ) + ∇ f 2 ( x k )) .
If they are conflicting, we can rewrite d k = − 12 ( v PC 1 + v PC 2 ) by extending v PC 1 and v PC 2 as the following equivalent form.
[Equation]
Then λ k = − ( 12 − ∇ f 1 ( x k ) ·∇ f 2 ( x k ) 2 ∥∇ f 1 ( x k ) ∥
[Equation]
Since PCGrad with more than three objectives should be presented as a protocol [48], we leave more details for the equivalent protocol with the general framework version in Appendix B.
Note that the composite weights λ k of PCGrad and CAGrad are not constrained in the probability simplex S m .
Hence, we here consider a more general assumption on the boundness of λ k for including various MOGM methods than bounding it in S m as MGDA.
Assumption 1.
For the general framework 2, there exists a finite constant B ∈ R , such that
[Equation]
all
[Equation]
n [Equation]
Remark.
The general framework also contains more recent approaches that are based on the fundamentals of the introduced methods, such as GradVac [44], IMTL-G [31].
Note that we focus on the optimization side for multi-objective problems, thus we provide the derivation of existing fundamental optimizers.
A vertical line of works that focus on neural architecture, such as RotoGrad [18], are not included in this work.
3.2 The Framework with Stochastic Gradients
MOO can be solved with gradient manipulation algorithms with full gradients with theoretical guarantees [11, 48, 30].
However, training with mini-batch gradient is much more practical for deep learning applications [38], while MOGM with stochastic gradients is largely underexplored.
Specifically, we only obtain noisy gradient feedback G k = (cid:0) g 1 ( x k , ξ 1 k ) , . .
g m ( x k , ξ mk ) (cid:1) that is corrupted by stochastic noise ξ ik , i = 1 , . .
, m .
Following 2, we have its stochastic version
[Equation]
[Equation]
Note that here λ k depends on stochastic gradients, not the exact ones.
We then make the following standard assumption similar to single objective setting [20, 14], which says that we have access to unbiased stochastic estimates of the gradients for all objectives.
Assumption 2.
Let { x k } nk =1 be the iterate sequence for any algorithm.
For all k = 1 , . . . , n , we have access to g 1 ( x k , ξ 1 k ) , . . .
, g m ( x k , ξ mk ) which are unbiased estimates of ∇ f 1 ( x k ) , . . . , ∇ f m ( x k ) , i.e. E ξ k (cid:2)
g i ( x k , ξ ik ) (cid:3) = ∇ f i ( x k ) , i = 1 , . .
, m. Further, each gradient variance is bounded by
E ξ k (cid:2)
∥ g i ( x k , ξ ik ) − ∇ f i ( x k ) ∥ 22 (cid:3) ≤ σ 2 , i = 1 , . . . , m .
To analyze MOGM in the stochastic setting, we state the following definition to determine a sequence generated by an iterative algorithm to asymptotic converge to Pareto critical and optimal.
Definition 3 ( Convergence condition )
(a) A sequence { x k } ∞ k =1 asymptotically converges to Pareto critical points if lim k →∞ E [min λ ∗ k ∈ S m ∥∇ F ( x k ) λ ∗ k ∥ ] → 0 .
(b) For convex multi-objective function F ( · ) .
A sequence { x k } ∞ k =1 asymptotically converges to Pareto optimal if lim k →∞ E [max x ∗ k ∈K min λ ∗ k ∈ S m ( λ ∗ k ⊤ F ( x k ) − λ ∗ k ⊤ F ( x ∗ k ))]
→ 0 .
−∇𝑓 !
−∇𝑓 "
PCGrad MGDA
CAGrad 𝔼[𝒅] (a) Inverse Direction 0.0 0.2 0.4 0.6 0.8 1.0 Iterations × 109.
f 1 ( x , y )
MGDACR-MGDA
PCGradCR-PCGrad
CAGradCR-CAGrad (b) Loss0.0 0.2 0.4 0.
0.8 1.0 Iterations × 10 5 9.
f 2 ( x , y )
MGDACR-MGDA
PCGradCR-PCGrad
CAGradCR-CAGrad (c) Loss0.0 0.2 0.4 0.
0.8 1.0 Iterations × 10 5 0.
D i s t an c e t
o P a r e t o
S e t
MGDACR-MGDA
PCGradCR-PCGrad
CAGradCR-CAGrad (
[Figure]
Optimal The definition (a) directly follows Proposition 1 (a) and the definition of Pareto critical.
The definition (b) is because x ∗ k is the minimum of λ ∗ k ⊤ F ( x ) , thus is Pareto optimal (Proposition 1 (b)).
Asymptotically converging to λ ∗ k ⊤ F ( x ∗ k ) implies that x k approaches Pareto optimal.
Note that the minimization operator of λ ∗ k is from the existence condition in Proposition 1 (b).
This definition can also be viewed as the stochastic version of metric function [42] and Pareto suboptimal gap [19].
The Non-convergence of Stochastic Multi-Objective Gradient Manipulation
With the problem setup of stochastic MOGM in the previous section, we now discuss fundamental flaws among the current gradient manipulation methods that are all designed and analyzed based on full-batch gradients.
We show that such batch algorithms fail to converge to a Pareto optimal solution even in a simple two-objective two-dimensional convex stochastic setting.
The main issue lies in the strong dependence of λ k on the received stochastic gradients G k .
Surprisingly, we find that combining such correlated λ k with the stochastic gradient yields a biased composite gradient E
[ d k ] = − E [ G k λ k ] that even conflicts with all full-batch gradients in expectation.
This inverse direction even causes the algorithms to inversely optimize all the objectives simultaneously.
An illustrative example.
Next, we provide a simple problem instance in which MGDA, PCGrad,
CAGrad can not converge to Pareto optimal.
Consider the following two-dimensional stochastic optimization setting over the domain K = { ( x, y ) | x ∈ [ − 1 , 1] , y ≥ 0 } , where p represents the probability of occurrence
[Equation]
The expected function is f 1 ( x, y ) =
( x + 3)
+ y 2 + 8 , f 2 ( x, y ) =
( x − 3)
+ y 2 + 8 .
Thus, by ignoring the constant term, the optimization goal is to minimize the distance towards ( − 3 , 0) and (3 , 0) simultaneously.
We can easily know that the Pareto set is the line segment { ( a, 0) | a ∈ [ − 1 , 1] } .
In the first step, by calculating the expected gradient of the point (0 , 0 . 4) , we find that all the gradient manipulation algorithms generate a vertical-up direction in expectation, as shown in Figure 1a, which moves the algorithm in the wrong direction away from Pareto optimal setting.
More seriously, the expected direction E [ d k ] = (0 , ϵ ) , ϵ > 0 is conflicting with both true gradients, suggesting that it inversely optimizes both objectives.
Furthermore, we could prove that the inverse direction problem happens with Pareto optimal points, which resists the algorithms to approach Pareto optimal.
Using the intuition above, we show that gradient manipulation algorithms can not converge to Pareto optimal for this setting.
Theorem 1.
There is a stochastic convex optimization problem for which MGDA, PCGrad, CAGrad do not converge to the Pareto optimal solution.
We relegate the proof to Appendix D. In the proof, we provide stronger support that at least in the range { ( x, y ) | x ∈ [ − 1 , 1] , y ∈
[0 , 0 . 05] } , the expected directions for the three typical algorithms all move the solution away from Pareto set.
This means there exists a constant gap for gradient manipulation algorithms to approach the Pareto optimal set.
Note that this gap is a sufficient condition to verify the non-convergence, and it can be much larger, as shown in Figure 1d.
Intuitively, the convergence gap is caused by the strong correlation between the composite weights λ k and the stochastic gradients G k , i.e., λ k is solely determined by G k , which could produce an inverse direction in the example above.
When the composite weights is independent of the stochastic gradients, the expected composite gradient E [ d k ] = − E [ G k λ k ] = −∇ F ( x k ) λ k will be a positive direction to approach the Pareto set.
Hence, methods like linear scalarization are able to attain a point in the Pareto optimal set [13], but are deprived of the practical benefits of gradient manipulation.
We would also like to emphasize that while the example of non-convergence is carefully constructed to demonstrate the problems in the above three typical algorithms more easily, such an issue also happens with more general Gaussian noise, as shown in Appendix G.2.
Hence, it is not unrealistic to suspect scenarios where such an issue can impair practical performance.
It is also reasonable to assume that this problem is especially aggravated in high dimensional settings, where the variance of the gradients is more significant.
Comparison with existing convergence results.
As introduced in Section 2.2, the optimal solution for the multi-task objective 1 m (cid:80) mi =
f i ( x ) must be Pareto optimal.
Therefore, the above nonconvergence analysis generates well with MOGM algorithms that optimize the averaging loss, such as CAGrad.
This result shows that the previous convergence analyses in the full-batch setting are all invalid in the stochastic setting.
To the best of our knowledge, there is only MGDA that has convergence analysis in the stochastic setting.
Liu et al.
[33] establishes the convergence of MGDA with stochastic gradients, but based on two unjustified assumptions.
The first is that the gradient variance asymptotically converges to 0.
The second is that the subproblem for λ k , denoted as function λ k ( g 1 , . . .
, g m ) , is Lipschitz continuous with right to each gradient.
The first assumption needs to gradually increase the batch size to full batch; however, the reliance on giant batch sizes is impractical for deep learning training [4].
The second one can significantly simplify the convergence analysis, but is actually not true as shown in the following proposition.
Proposition 2.
Subproblem λ k ( · ) is not Lipschitz continuous with right to each gradient for MGDA, i.e., there exist no finite β such that ∥ λ k ( g 1 , . .
, g i , . . .
, g m ) − λ k ( g 1 , . .
g ′ i , . .
g m )
∥ ≤ β ∥
g i − g ′ i
∥ for i = 1 , . . .
, m and gradient g 1 , . .
, g i , g ′ i , . . . , g m .
Similar results can be proved for PCGrad and CAGrad, and we relegate the proof to the appendix D.
Correlation-Reduced Stochastic Multi-Objective Gradient Manipulation
In this section, we develop a new principled exponential moving average mechanism on the composite weights λ k , and provide its theoretical benefit for fixing the convergence issue for MOGM algorithms.
Our aim is to devise a new strategy with guaranteed convergence, while preserving the practical benefits of MOGM without introducing additional computation and memory overhead.
Algorithmic insight.
To understand the design of the mechanism, recall that the main issue of the non-convergence for MOGM lies in the strong correlation between the composite weights λ k and the stochastic gradients G k , which could cause the composite gradient E
[ d k ] = − E [ G k λ k ] inversely optimizing all the objectives in expectation.
It is natural to consider reducing such correlation, which can be bounded by the following lemma.
Lemma 1.
Under Assumption 1, 2.
The correlation between λ k and G k can be bounded as ∥
[Equation]
, where V [Equation]
=
[Equation]
From Lemma 1, we know that the correlation can be reduced by reducing the variances of λ k .
From this intuition, we can modify MOGM algorithms to reduce such correlation by regularizing the variation of λ k .
Correlation-reduced MOGM (CR-MOGM) adopts a momentum mechanism with coefficient α k on the update of composite weights as Line 5 in Algorithm 1, which is shown to be able to reduce variance as
Lemma 2.
Under Assumption 1, λ k variance for Algorithm 1 is bounded V ξ k
[ λ k ] ≤ m 2 B 2 (1 − α k ) 2 .
This result shows that the correlation can be controlled by choosing α k closed enough to 1.
With a reduced correlation, the wrong direction generated by MOGM could be corrected to a positive direction, and thus the algorithm would be able to converge.
Note that the momentum on λ k only requires an additional negligible computation cost.
On the other hand, the proposed exponential moving average mechanism also preserves the advantage of MOGM by carefully setting the momentum parameter α k to be 0 and asymptotically convergent to 1 .
Intuitively, CR-MOGM performs similarly to vanilla MOGM at the initial stage, and approximates like fixed linear scalarization in the late stage.
Initially, the stochastic noise occupies a relatively small component of the stochastic gradient.
Thus, methods like MGDA, PCGrad, CAGrad can search for good common descent directions.
When the number of iterations increases and the solution approaches the Pareto optimal, the impact of the stochastic noise stands out, causing the inverse direction problem for MOGA as in the previous section.
At this time, α k becomes larger, maintaining relatively stable λ
k eliminates the strong correlation and guarantees the convergence.
Convergence analysis.
We next provide convergence analysis for CR-MOGM 1 concluding all algorithms based on the general framework 3.
Before that, we introduce basic assumptions.
Assumption 3.
f 1 ( x ) , . .
f m ( x ) are all differentialable, H -Lipschitz and L -smoothness, suggesting that for all x , y ∈ K , i ∈
[ m ] , it holds ∥∇ f i ( y )
∥ ≤ H and ∥∇ f i ( x ) − ∇ f i ( y )
∥
≤ L ∥ x − y ∥ .
Lipschitz and smoothness are commonly used in complexity analysis for stochastic gradient algorithms [20, 14, 12, 4].
Based on Definition 3, we prove the following key results for CR-MOGM.
Theorem 2 ( Convex ) .
Under Assumption 1, 2, 3.
For the sequence x 0 , x 1 , . . .
, x n generated by
Algorithm 1, we assume objective functions f 1 ( x ) , . . .
, f m ( x ) are all convex with bounded optimal values as f 1 ( x ∗ ) , . . .
, f m ( x ∗ ) ≤ F for x ∗ in Pareto set, and the distance from sequence to Pareto set is bounded, i.e., ∥ x k − x ∗ ∥ ≤ D .
Set 0 ≤ η n ≤ . . .
≤ η 1 ≤ 1 /mLB, α k ∈ (0 , 1] for k = 1 , . .
. , n in Algorithm 1, it achieves
[Equation]
Theorem 3 ( Non-convex ) .
Under Assumption 1, 2, 3. Under Assumption 1, 2, 3.
For the sequence x 0 , x 1 , . . .
, x n generated by Algorithm 1, we assume objective functions
f 1 ( x k ) , . . .
, f m ( x k ) ≤ F are all bounded for k = 1 , . . . , n .
Set 0 ≤ η n ≤ . . .
≤ η 1 ≤ 1 /mLB and α k ∈ (0 , 1] in Algorithm 1, it achieves
[Equation]
Remark.
Note that our theoretical results imply that CR-MOGM is convergent in averaging by choosing suitable η and α .
Such an averaging analysis scheme is adopted by CAGrad [30], and is often used in stochastic optimization [41, 15, 4], e.g. Adagrad [28], Adam [24, 38].
It can be transformed into the traditional version, as shown in Appendix E.
The above results falls as an immediate corollary of the above Theorem.
4 Iterations × 100.
T r a i n i ng
Lo ss - L
MGDACR-MGDA
PCGradCR-PCGrad
CAGradCR-CAGrad (
[Figure]
Training Loss-L    4 Iterations × 100.
T r a i n i ng
Lo ss - R
MGDACR-MGDA
PCGradCR-PCGrad
CAGradCR-CAGrad (
[Figure]
Training Loss-
[Figure]
4 Iterations × 100.
T e s t
i ng
Lo ss - L
[Figure]
PCGrad
CAGradCR
[Figure]
4 Iterations × 100.
T e s t
i ng
Lo
[Figure]
R
MGDACR
[Figure]
PCGradCR
[Figure]
PCGrad
CAGradCR
[Figure]
[Table]
MGDA
[Table]
[Table] [Table]
Corollary 1.
Set 1 ≥ η k
= O (1 / √ k ) , α k = max { 1 −
η k /η 1 , 1 − η k /η 1 ( k (cid:80) mi =1 | λ ik − 1 − ˆ λ ik | ) } in
Theorem 2,3, Algorithm 1 converges with O ( n 1 / 2 ) , O ( n 1 / 4 ) rates in the convex and non-convex case.
This result indicates that Algorithm 1 has a convergence rate that matches the one in the singleobjective stochastic setting [7, 2].
It demonstrates the efficacy of the proposed momentum mechanism.
Experiments
In this section, we present empirical results on both synthetic with convex objectives and real-world datasets for multi-task learning with deep neural networks, representing convex and nonconvex settings, respectively.
We apply CR-MOGM to MGDA, PCGrad, CAGrad, and use CR- to distinguish whether it adopts our approach.
We conduct experiments to answer the following questions:
Q1:
Does our approach fix the non-convergence and improve the optimality for MOGM algorithms?
Q2:
Does our approach consistently improve the practical performance of MOGM in deep learning?
Q3:
Does the non-convergence issue demonstrated with the special case still exists in practice?
Q4:
Is our approach flexible to be implemented and agnostic to models as well as optimizers?
Synthetic experiments.
To demonstrate whether CR-MOGM fixes the convergence issue for gradient manipulation algorithm, we first consider the same simple convex setting as the example 4 in Section 4.
We initialize x 0 = (0 , 0 . 4) and use SGD as the optimizer.
To enable fair comparison, the adaptive stepsize is set with η k = 0 .
006 / √ k for all the algorithms, and the momentum parameter for smoothing the composite weights is set with the suggested parameter setting in Corollary 1.
Figure 1 illustrates the multi-objective losses and distance to the Pareto set for this problem.
We first note that both the two objective losses of MGDA, PCGrad, CAGrad are significantly larger than their counterparts applied with the proposed mechanism.
Furthermore, the losses are even increasing at the initial stage as a result of the inverse direction issue demonstrated in Figure 1a.
It moves x k away from the Pareto set, which unfortunately causes an increasing suboptimal gap.
In contrast, applying
CR-MOGM enhances the algorithms to converge to Pareto optimal.
This positively answers Q1 .
Multi-task classification.
For multi-task classification scenarios, we follow the experiment setup from Sener et al.
[40], and consider the MultiMNIST dataset [39] with 60K examples.
MultiMNIST contains two tasks: classifying the digit on the top-left (task-L) and classifying the digit on the bottom-
[Table]
We report the train and test losses in Figure 2.
We can see that algorithms with λ momentum perform better than original ones with respect to both train and test losses.
We also observed that λ momentum consistently improves the accuracy for all the objectives in Table 1.
We hence positively answer Q2 .
In particular in Table 2, we observe that the loss of vanilla MGDA becomes significantly larger when the batch size is smaller, while CR-MGDA appears not sensitive to the batch size.
The possible reason is that the convergence issue of MOGM becomes more severe with larger stochastic noise, which severely degrades the performance of MOGM.
Since CR-MOGM is proven to converge to the
Pareto optimal/critical, it is much more robust to stochastic noises.
Thus, in this regard, our method is more practical for real-world applications.
We hence positively answer Q3 .
Semantic segmentation For semantic segmentation scenarios, we follow the experiment setup from
Yu et al.
[48] and Liu et al.
[30], and consider the CityScapes.
CityScapes similarly contains two tasks: 7-class semantic segmentation and depth estimation.
We combine algorithms with MTAN [32] based on SegNet architecture [1].
We use adam [24] as the optimizer.
All parameters are tuned from vanilla MOGM algorithms, and the correlation-reduced counterparts follow the same parameter setting to enable a fair comparison.
The momentum parameter is set with the suggested value.
We report mean Intersection-Over-Union (mIoU), pixel accuracy (Pix Acc), and absolute/relative (abs/rel) errors in Table 3.
We observe comparable or better performance when adopting our approach with the three gradient manipulation methods.
In particular, the improvements for MGDA and CAGrad are more significant than PCGrad, possibly due to the special gradient manipulation rule that PCGrad does not alter the gradient when they are not conflicting, then the non-convergence issue of PCGrad could be less severe.
Combining the above results, we demonstrate that our methods work for different models and optimizers.
We hence give a positive answer to Q4 .
Conclusions
In this work, we study multi-objective gradient manipulation in the stochastic mini-batch setting and identify a fundamental flaw in algorithms designed under the exact full-batch setting.
Through an illustrative example, we rigorously prove the non-convergence issue of typical methods, including
MGDA, PCGrad, and CAGrad.
In particular, we fix this problem by slightly changing these algorithms by an exponential smoothing mechanism without introducing additional computational or memory overhead.
We prove the convergence of the resulting algorithms and show that they yield consistent empirical performance improvement.
Our analysis is general and can be extended to analyze combining MOGM algorithms with other stochastic optimizers such as Adam and Adagrad.
Limitations.
This paper discusses the typical MOGM algorithms with strict convergence guarantees, i.e., MGDA, PCGrad, and CAGrad.
Other empirical-driven methods without rigorous theoretical guarantees, such as RotoGrad [18], are too expensive to be included in our analysis.
We believe that the convergence issue in that line of literature will be an excellent future work to explore in depth.
Acknowledgement
This work was supported by the National Key Research and Development Program of China No.
2020AAA0106300 and the National Natural Science Foundation of China (No. 62050110, No.
61936011, No. 62102222).
This work was supported by Ant Group through Ant Research Intern
Program.
We would like to thank Tianyu Liu, Xin Wang, and Ziwei Zhang for their advices on paper writing.
Convergence
of Stochastic
Multi-Objective Gradient
Alteration and Beyond”
We organize the supplemental materials as follows.
Section A provides a detailed comparison between weak Pareto optimal and Pareto optimal.
Section B provides the derivation of PCGrad with more than two objectives to our general framework.
Section C discusses more details of the convergence analysis framework in Definition 3.
Section D supplements the theoretical results on the non-convergence for
MOGA algorithms.
Section E supplements the convergence results for both convex (Theorem 2) and non-convex (Theorem 3) cases.
In particular, we discuss the averaging scheme of the convergence theorem in more detail, and provide a way to transform it to traditional convergence using a typical uniformly sampling technique in stochastic convergence analysis [4, 26, 7, 22].
Section F provides the convergence analysis for the cases with strongly convex functions.
Section G introduces additional details of the experiments, and provides an additional simulation result with Gaussian noises.
A Detailed Comparison between Weak Pareto Optimal and Pareto Optimal
In this section, we give a rigorous comparison between the definitions of Pareto optimal and weak
Pareto optimal, where the latter is defined as
Definition 4 (Weak Pareto optimal) .
A solution x ∗ ∈ K is called weak Pareto optimal if there is not x ∈ K such that F ( x ) < F ( x ∗ ) , i.e. f i ( x ) ≤ f i ( x ∗ ) for all i = 1 , . . .
, m .
By the Definition 1 and Definition 4, it can be easy to know that Pareto optimal points are always weak Pareto optimal, and the converse is not always true.
Take a two-objectives example, the minimal points for f 1 are all Pareto weak optimal, but only ones that are also optimal for f 2 are Pareto optimal.
Furthermore, if we assume f 1 has a unique minimizer, which is then Pareto optimal.
In rigorous words, we state below that relation among Pareto optimal and weak Pareto optimal, as a rigorous version for Proposition 1 (b).
Proposition 3 (Theorem 5.13 and Lemma 5,14 in [17]) .
For MOO problem 1, the following statements hold
: (a) For convex objectives F ( · ) .
If there exists λ ∈ S m such that x ∗ = arg min x ∈K λ
T F ( x ) , then x ∗ is weak Pareto optimal.
( b)
Further if x ∗ is the unique minimizer then x ∗ Pareto optimal
For ease to understand, previous studies [11, 33, 30] do not distinguish the detailed difference between the above definitions.
This paper also follows this presentation trick.
B More Details of the General Framework for MOGA
PCGrad with more than 2 objectives.
Specifically, the PCGrad is initialized by v PC i = ∇ f i ( x k ) .
When subtracting the conflicting component, i.e., v PC i = v PC i − v PC i ⊤ ∇ f j ( x k ) ∥∇ f j ( x k ) ∥ 2 ∇ f j ( x k ) , it is equivalent to subtracting a weight v PC i ⊤ ∇ f j ( x k ) ∥∇ f j ( x k ) ∥ 2 ∇ f j ( x k ) for gradient v j if using the framework 2.
Provided with such equivalence, d k = − 1 m (cid:80) mi =1 v PC i is equivalent to d k = −∇ F ( x k )
λ k using Algorithm
1: Input: ∇ f i ( x k ) , i = 1 , . . .
, m 2: Initialize: v i = ∇ f i ( x k ) , v PC i = v i , λ i = 1 for i = 1 , . . .
, m 3: for i ∈ [ m ] do 4: for j uniformly ∼ [ m ] \ i in random order do 5: if v PC i ⊤ ∇ f j ( x k ) < 0 then 6:
Set v PC i = v PC i − v PC i ⊤ ∇ f j ( x k ) ∥∇ f j ( x k ) ∥ 2 ∇ f j ( x k ) 7:
Update λ j = λ
j − v PC i ⊤ ∇ f j ( x k ) ∥∇ f j ( x k ) ∥8: end if 9: end for 10: end for 11: return λ
C More Details of the Convergence Analysis Framework
In this section, we provide more details about our stochastic convergence analysis framework.
We first offer proof details of the Proportion 1, and then generalize previous analysis based on it.
Finally, we give a discussion for Definition 3 that is the stochastic convergence framework based on the previous full-batch setting.
C.1 Proof of Proportion
Proof.
The Pareto criticality can be equivalently defined as: a x ∗ ∈ K is Pareto critical, if min λ ∗ ∈ S m ∥ (cid:80) mi =1 λ ∗ i ∇
f i ( x ∗ ) ∥
= 0 , where S m = { ( λ 1 , . .
λ m ) | (cid:80) mi =
λ i = 1 , λ ∈ [0 , 1] } denotes the probability simplex.
Therefore, if there exist a λ ∈ S m , such that 0 ≤ ∥ (cid:80) mi =1 λ i ∇ f i ( x ∗ ) ∥
= 0 , then min λ ∗ ∈ S m ∥ (cid:80) mi =1 λ ∗ i ∇ f i ( x ∗ )
∥
≤ ∥ (cid:80) mi
=
λ i ∇ f i ( x ∗ ) = 0 .
We thus prove Proposition (a) , and Proposition (b) directly comes from Proposition 3.
C.2 More Details of Previous Convergence Analysis in Full-batch Gradient Setting
Previous convergence analyses in full-batch gradient setting, including MGDA [11, 42], PCGrad [48],
CAGrad [30] can be analyzed based on Proposition 1 that reflects Pareto optimality and criticality, which induces the following convergence analysis framework.
Definition 5.
a)
A sequence {
x k }
∞ k
=
1 asymptotically converges to Pareto critical points
if lim k →∞ min λ ∗ k ∈ S m ∥∇ F ( x k ) λ ∗ k ∥ → 0 .
(b) For convex multi-objective function F ( · ) .
A sequence { x k } ∞ k =1 asymptotically converges to Pareto optimal if lim k →∞ max x
[Equation]
k ∈K
[Equation]
λ ∗ k ∈ S m
( λ ∗ k ⊤ F
[Equation]
x
[Equation] [Equation]
λ
[Equation]
k ⊤ F ( x ∗ k ))
→ 0 .
(a) can be reviewed as the limit for attaining the criticality condition (the equivalent definition in [30]).
(b) is equivalent to that there exists a composite weight λ ∗ k ∈ S m such that λ ∗ k ⊤ F ( x k ) − min x ∗ k ∈K λ ∗ k ⊤ F ( x ∗ k )
→ 0 , which is essentially the limit for attaining Pareto optimal condition in
Proposition 1.
It can be also reviewed the the convergence for multiobjective metric function [42].
We next briefly introduce the analysis of MGDA [11, 42] as an typical example that provides both nonconvex and convex convergence.
It proves that the norm of min-norm direction min λ ∗ k ∈ S m ∥∇ F ( x k ) λ ∗ k
∥ is asymptotically convergent to zero with an order of O (1 / √ n ) , which is reviewed as the convergence rate with non-convex functions.
Theorem 4 (Theorem 3.1 in [11]) .
For MGDA algorithm updating with stepsize η ≤ 1 /L
[Equation]
This result also implies that the composite gradient
d k = ∥∇ F ( x k ) λ ∗ k ∥ for MGDA asymptotically vanishes, which indicates that the sequence x k iterated by gradient descent using d k limits to a critical point x ∗ .
If the multi-objective function is convex for each objective, such a critical point x ∗ is also weak Pareto optimal [43].
It is also not difficult to verify that if x ∗ is a Pareto stationary point, then it satisfies ∇ F ( x ∗ ) λ ∗ = 0 for one λ ∗ ∈ S m .
If the functions are convex, this meets the optimal condition of the composite function λ ∗⊤ F ( x ∗ ) .
And by Proposition 1, we then know that x is weak Pareto optimal.
It can prove that (cid:80) nk
=1 λ ∗ k ⊤ F ( x k ) − (cid:80)
nk
=1 λ ∗ k ⊤ F ( x ∗ ) ≤
O (1) .
Since in the batch-gradient setting, d k is the common descent direction such that F ( x k ) is decreasing with respect to k .
Thus it has
[Equation]
Therefore, we have the following theorem
Theorem 5 (Theorem 4.1 in [11]) .
For MGDA algorithm updating with stepsize η ≤ 1 /L
[Equation]
Where the averaging weight λ ∗ k n is also in S m .
Since in the proof, the comparing point x ∗ can be generated to arbitrarily chosen u ∈ K [43].
We can further generate the result with (cid:16) (cid:80) nk
[Equation]
1 λ
[Equation]
n
[Equation]
− (cid:16) (cid:80) n
[Equation]
1 λ
[Equation]
k n (cid:17) ⊤ F ( u ) ≤
O (1 /k ) .
By letting u = arg min x ∈K (cid:16)
[Equation]
(cid:17) ⊤ F ( x ) , it finally has
[Equation]
Similarly, PCGrad [48] is proved to converge at a Pareto critical point (Theorem 1 in [48]).
CAGrad
[30] generalizes the analysis of MGDA, and shows that it converges to the Pareto critical point that tends to be the critical point of the average loss when c ∈ [0 , 1) .
These convergence results can all be analyzed under the Pareto convergence framework as Definition 5.
C.3 More Discussion for Definition
Definition 3.
(a) A sequence { x k } ∞ k =1 asymptotic converges to Pareto critical points if
[Equation]
(b) For convex multi-objective function F ( · ) .
A sequence { x k } ∞ k =1 asymptotic converges to Pareto optimal if
[Equation]
Definition 3 is a stochastic version for Definition 5.
Similarly, (a) can be reviewed as the expected limit for attaining the criticality condition.
(b) is equivalent to that there exists a composite weight λ ∗ k ∈ S m such that E [ λ ∗ k ⊤ F ( x k )
− min x ∗ k ∈K λ ∗ k ⊤ F ( x ∗ k )]
→ 0 , which means λ ∗ k ⊤ E [ F ( x k )]
→ min x ∗ k ∈K λ ∗ k ⊤ F ( x ∗ k ) , suggesting that the expected function value E [ F ( x k )] are optimal.
Thus,
Definition 3 is indeed derived from Pareto optimality and criticality.
D Missing Proofs for the Non-convergence of MOGA
In this section, we give detailed proof for the non-convergence for MGDA, PCGrad, CAGrad with the example introduced in Section 4.
We then prove the non-Lipschitz of the composite weight for the above MOGA algorithms.
D.1 Proof of Theorem
Proof.
Recall the two-dimensional stochastic optimization setting over the domain K = { ( x, y ) | x ∈ [ − 1 , 1] , y ≥ 0 }
[Equation]
The expected function is f 1 ( x, y ) =
( x + 3)
[Equation]
+ y
[Equation]
+ 8 , f 2 ( x, y ) =
( x − 3)
+ y 2 + 8 .
Therefore, the optimization goal is to minimize the distance towards ( − 3 , 0) and (3 , 0) simultaneously.
We can easily know that the Pareto set is the line segment { ( a, 0) | a ∈ [ − 1 , 1] } .
We can calculate the stochastic gradient in the above setting as ∇
[Equation]
Next, we prove that for all point in domain { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the expected composite directions
E
[ d k ] =
( d 1 k , d 2 k ) of MGDA, PCGrad, CAGrad have positive value in y-axis, i.e. d 2 k > 0 , which forces the algorithms to move away from the Pareto set.
Lemma 3 (Sener et al.
The min-norm solver has closed form solution for two-objective case.
Specifically, for g 1 , g 2 , the composite weight λ for g 1 can be calculated as
[Equation]
where the operator [ a ] + = max { a, 0 } .
Lemma 4.
For any point in domain { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , MGDA has a expected composite directions
E
[ d k ] =
( d 1 k , d 2 k ) such that d 2 k > 0 .
Proof.
We adopt simple enumeration method to calculate the expected direction.
Case 1: when the received gradients are
[Equation]
From 5, we can get λ
=
(1 − a ) / 2 , then the direction for ( a, b ) is
[Equation]
Case 2: when the received gradients are
[Equation]
From 5, we can get λ
=
(5 − a ) / 10 , then the direction for ( a, b ) is
[Equation]
Case 3: when the received gradients are
[Equation]
From 5, we can get λ
= ( − 3 a + 2 b + 19) / 26 , then the direction for ( a, b ) is
[Equation]
Case 4: when the received gradients are
[Equation]
From 5, we can get λ
= ( − 3 a − 2 b + 7) / 26 , then the direction for ( a, b ) is
[Equation]
By summing up the above cases and multiply the probability, we have
[Equation]
Since we know that b ≤ 0 .
05 , we then know d 2 k ≥ 12 − 22 ∗ 0 .
05= 10 . 9 13 > 0 .
We thus prove the
Lemma.
Lemma 5.
For the point in domain { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , PCGrad has a expected composite directions
E
[ d k ] =
( d 1 k , d 2 k ) such that d 2 k > 0 .
Proof.
Similar with the proof of MGDA, we also adopt simple enumeration method to calculate the expected direction.
Case 1: when the received gradients are
[Equation]
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients has not conflicting, then we know the y-axis value of the composite gradient for ( a, b ) is 4 − 2 b .
Case 2: when the received gradients are
[Equation]
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients consistently conflict with each other, then we know the y-axis value of the composite gradient for ( a, b ) is
[Equation]
Case 3: when the received gradients are
[Equation]
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients consistently conflict with each other, then we know the y-axis value of the composite gradient for
[Equation]
[Equation]
Case 4: when the received gradients are
[Equation]
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients consistently conflict with each other, then we know the y-axis value of the composite gradient for ( a, b ) is
[Equation]
Sum up the above, it is easy to know d 2 k is a increasing function for b , and is creasing for a ∈ [ − 1 , 0] and decreasing for a ∈ [0 , 1] .
Plug in a = ± 1 , b = 0 .
05 , we can get d 2 k ≥ 0 . 2 > 0 .
Lemma 6.
For the point in domain { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , CAGrad with c ∈ [0 . 8 , 1] has a expected composite directions
E
[ d k ] =
( d 1 k , d 2 k ) such that d 2 k > 0 .
Proof.
Similar with the proof of MGDA, we also adopt simple enumeration method to calculate the expected direction.
Case 1: when the received gradients are
[Equation]
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients has not conflicting, then we know the y-axis value of the composite gradient for ( a, b ) is − 2 b +4+2 c
(cid:112) a 2 + ( b − 2) 2 ≥ − 2 ∗ 0 .
+ 4 + 2 c (cid:112) 0 2 + (0 .
− 2) 2 = 3 .
8 c + 3 . 9 ≥ 6 . 94 .
Case 2: when the received gradients are
[Equation]
0.4 0.6 0.8 x 0.
y
Expected Direction (a) MGDA 1.0 0.8 0.6 0.4 0.2 0.0 0.2 0.4 0.6 0.8 x 0.
y
Expected Direction (b) PCGrad 1.0 0.8 0.6 0.4 0.2 0.
0.2 0.4 0.6 0.8 x 0.
y
Expected Direction
[Figure]
We can observe that this biased direction can move algorithms away from the Pareto optimal set, which resists the algorithm to converge.
Additionally, the stationary points for all three algorithms does not lies near the Pareto set.
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients consistently conflict with each other, then we know the y-axis value of the composite gradient for ( a, b ) is − 2 b −
[Equation]
Case 3: when the received gradients are
[Equation]
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients consistently conflict with each other, then we know the y-axis value of the composite gradient for ( a, b ) is − 2 b + c √ 22 (cid:112) (
[Equation]
Case 4: when the received gradients are
[Equation]
It is easy to verify that for { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } , the two gradients consistently conflict with each other, then we know the y-axis value of the composite gradient for ( a, b ) is − 2 b +
[Equation]
[Equation]
Sum up the above, we can get d 2 k ≥ 2 . 5 > 0 .
Figure 3 shows the vector field for the composite direction calculated by MGDA, PCGrad, CAGrad, which aligns with the above Lemmas that indicate the expected direction is inverse to the Pareto set.
This problem implies that E [ x k +1 ] is more away from Pareto optimal than E [ x k ] if x k lies in domain { ( a, b ) | a ∈ [ − 1 , 1] , b ∈ [0 , 0 . 05] } .
Thus the expected suboptimality gap is always > 0 , which implies that MGDA, PCGrad, CAGrad do not converge to the optimal solution.
In addition, we could also infer from the figure that, the three MOGA algorithms are stationary around (0 , 0 . 4 0 . 6) , which represents a large expected suboptimality gap in this case.
D.2 Proof of Proposition
Proof.
The proposition can be proved by the following simple 2 objectives case, where we are provided with g
= (1 , 0) , g 2 = (1 − ε, 0) , g ′ 2 =
(1 + ε, 0) where ε << 1 .
For this function, the min-norm direction is g 2 for the first case, while is g 1 for the second case, which means λ ( g 1 , g 2 )
=
(0 , 1) and λ ( g 1 , g ′ 2 ) =
However, we know that g 2 , g ′ 2 only have 2 ε gap.
When ε limits to 0, there exists no finite β to bound ∥ λ ( g 1 , g 2 )
− λ ( g 1 , g ′ 2 ) ∥ / ∥ g 2 − g ′ 2 ∥ .
Generalize to PCGrad.
This property can be generalized to PCGrad, we next provide another counterexample to prove it.
Proof.
Suppose we are provided with g 1 = (1 , 0) , g 2 = ( − ε, 0) , g ′
= ( ε, 0) , ε << 1 .
For this function, g 1 conflicts with g 2 while aligned with g ′ 2 .
This makes λ ( g 1 , g 2 ) =
( 12 + ε 2 , 12 + 12 ε ) and λ ( g 1 , g ′ 2 ) =
However, we know that g 2 , g ′ 2 only have 2 ε gap.
When ε limits to 0, there exists no finite β to bound ∥ λ ( g 1 , g 2 )
− λ ( g 1 , g ′ 2 ) ∥ / ∥ g 2 − g ′
∥ = (cid:113) 1 + 1 ε 4 / 4 .
Generalize to CAGrad.
This property can also be generalized to CAGrad with parameter c ∈ (0 , 1 / 4) with the similar example with PCGrad.
When c ∈ [1 / 4 , + ∞ ) , CAGrad performs more like MGDA, it is not difficult to generalize the example of MGDA.
Proof.
The proposition can be proved by the following simple 2 objectives case, where we are provided with g
= (1 , 0) , g 2 = ( − ε, 0) , g ′ 2 =
( ε, 0) , ε << 1 .
For this function, g 1 conflicts with g 2 while aligned with g ′ 2 .
This makes λ ( g 1 , g 2 ) =
( 1 − c (1 − ε ), 12 ) and λ ( g 1 , g ′ 2 ) =
( 1+ c (1+ ε ), 12 ) .
However, we know that g 2 , g ′ 2 only have 2 ε gap.
When ε limits to 0, there exists no finite β to bound ∥ λ ( g 1 , g 2 )
− λ ( g 1 , g ′ 2 ) ∥ / ∥ g 2 − g ′
∥ = c/ε .
E Missing Proofs for the Convergence of CRMOGA
We here give a rigorous convergence analysis of CRMOGA in the stochastic (mini-batch) gradient setting.
We first fulfill the proofs for Lemma 1 and Lemma 2.
Before providing the final proof, we would like to introduce some useful and insightful lemmas that reflect the algorithmic property and can be used for the final proof.
Specifically, Appendix E.4 and E.5 decompose the convex and non-convex convergence bounds into sub-terms bounded by lemmas from Appendix E.3.
E.1 Proof of Lemma
Lemma 1.
Under Assumption 1, 2.
The correlation between λ k and G k can be bounded as ∥ E [ G k λ k ] − E [ G k ] E
[ λ k ] ∥ 22 ≤ mσ 2 V ξ k ( λ k ) , where V ξ k ( λ k ) =
E ξ k [ ∥ λ k − E ξ k
[ λ k ] ∥ 22 ] .
Proof.
Denote g i ( x k , ξ ik ) =
g i ( x k ) , i ∈ [ m ] for simplicity.
By the unbiasness of each stochastic gradient E [ g i ( x k )]
= ∇ f i ( x k ) for each objective, we have
[Equation]
The last equation is by the fact that
[Equation]
where the first equation is by the linearity of expectation, and the second one is by the unbiasness of each stochastic gradient.
Further, by the fact that ∥ E [ a ] ∥ ≤ E [ ∥ a ∥ ] and the triangle inequality for the l 2 norm, we know that ∥ E [
[Equation]
Where, since 2 ab ≤ βa 2 + 1 β b 2 for β > 0 , we can get
[Equation]
Plug in, we obtain
[Equation]
The last equation is by the definition of V ξ k .
Here we let β k
=
[Equation]
and apply all the above result, we finally get
[Equation]
We thus prove the Lemma.
Remark.
Lemma 1 suggests that the correlation between the composite weights and stochastic gradients is bounded by both the variance of composite weights λ k and multiple stochastic gradients g i ( x k ) .
Therefore, there are two effective ways to decrease this correlation: reduce the variance of λ k , or the variance of g i ( x k ) .
For the latter way, one can adopt variance reduction framework [20, 4] for stochastic optimization.
However, it [20] requires global gradients that may be computational expensive, or slows down the convergence rate [4].
For the former one, the momentum mechanism on λ k is a simple and effective way, which enhances both theoretical and empirical results without additional computational cost.
E.2 Proof of Lemma
Lemma 2.
Under Assumption 1, λ k variance for Algorithm 1 is bounded V ξ k
[ λ k ] ≤ m 2 B 2 (1 − α k ) 2 .
Proof.
By the definition of Algorithm 1, we know that
[Equation]
From the fact that E [ ∥ X − E
[ X ] ∥ 22 ] ≤ E [ ∥ X − a ∥ 22 ] for any a , we can get
[Equation]
The last inequality is by the fact 0 ≤ λ ik , ˆ λ ik ≤ B, i = 1 , . . .
, m .
E.3 Lemmas needed for Proving Thm2/Thm
This subsection presents several useful Lemmas for proving Theorem 2 and Theorem 3.
Lemmameasures the expected gap between − (cid:80) mi
=1 λ ik ∇ f i ( x k ) and d k .
The following Lemma bounds the dot production between them, which measures the local decrease of the composite functional value.
Lemma 7.
Under Assumption 1, 2, 3, for any gradient descent algorithm updated with composite gradient G k λ k , we have the following inequality
[Equation]
[Table]
Proof.
We first decompose the term into
[Equation]
The first term can potentially corrupt the effectiveness of optimization, and the second term measures the descent value.
We next bound the first term.
By definitions and decomposition, we get
[Equation]
We then bound each term individually.
By Assumption 1 and Assumption 3, we know that λ ik ∈ [0 , B ] , ∥∇ f i ( x k ) ∥
≤ H, ∀ i = 1 , . . .
, m .
By Cauchy–Schwartz inequality, we further know that for the term A term A = E ξ k
[Equation]
k ))
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:35) ≤
[Equation]
=
[Equation]
=
[Equation] [Equation]
( x
[Equation]
i ( x k ))
(cid:13)(cid:13) (cid:13) (cid:13)(cid:13)(cid:35)
[Equation]
ξ k
[Equation]
m (cid:88) i =
BH
[Equation]
=
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
≤
[Equation]
(cid:35) .
Similar with the proof in Lemma 1, by the fact that ab ≤ 12 β k a 2 + β k 2 b 2 for any β k > 0 , and by the linearity of expectation, we can get term A ≤ mBH 2 β k m (cid:88) i =
E ξ k (cid:2) | λ ik − E [ λ ik ] | 2 (cid:3) + mBHβ km (cid:88) i =
E ξ k (cid:2) ∥∇ f i ( x k
[Equation]
By setting β k = (cid:112) V ξ k
[ λ k ] / (cid:113)(cid:80) mi
=1 V ξ k
[ g i ( x k )] , we have term A ≤ mBH (cid:118)(cid:117)(cid:117)(cid:116)
[Equation]
m (cid:88)
[Equation]
V [ g i ( x
[Equation]
With similar tricks, we have for the term B term
[Equation]
=
[Equation]
k
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
=
[Equation]
[Equation]
The first inequality is by Cauchy–Schwarz inequality.
The second one is by the triangle inequality of l 2 norm, we have ∥ (cid:80) mi =1 ( λ ik − E [ λ ik ]) ∇ f i ( x k ) ∥ 2 ≤ (cid:80) mi =1 ∥ ( λ ik − E [ λ ik ]) ∇ f i ( x k ) ∥
= (cid:80) mi =1 | λ ik − E [ λ ik ] |∥∇
f i ( x k ) ∥ 2 .
With Assumption 3, we know (cid:80) mi =1 | λ ik − E [ λ ik ] |∥∇
f
i ( x k ) ∥ 2 ≤ H (cid:80) mi =1 | λ ik −
E [ λ ik ] | .
The third one is from the fact that ab ≤ 12 β k a 2 + β k 2 b 2 .
The forth one is also from CauchySchwartz inequality.
The last one is from Assumption 1 that states E [ λ ik ] ≤ B .
Further by setting β k = (cid:112) V [ λ k ] /B
(cid:112)(cid:80) mi =1 V [ g i ( x k )] , we have term B ≤ (
[Equation]
[Equation]
(cid:118) (cid:117) (cid:117)(cid:116) V [ λ k ] m (cid:88) i =
V [ g i ( x k )] ≤ mBH (cid:118) (cid:117) (cid:117)(cid:116) V [ λ k ] m (cid:88) i =
V [ g i ( x k )] .
For term C, by the fact that only g i ( x k ) has randomness, we get term C = E ξ k   (cid:32) m (cid:88) i =
E
[Equation]
( x k ) (cid:33) ⊤ (cid:32) m (cid:88) i =
[Equation]
By summing up the above results
[Equation]
we obtain
[Equation]
Plug in the first decomposition in the beginning, we finally get
[Equation]
E ξ k   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) m (cid:88) i = λ
ik ∇ f i ( x k ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)    .
Remark.
The result from Lemma 7 indicates that the local decrease for expected composite loss
E [ λ k ] T F ( x ) is bounded by 2 mBH
[Equation]
(cid:104)(cid:13)(cid:13)(cid:80)
[Equation]
The first term consists of the variances for composite weights and multiple gradients, and the second term is the negative norm of the effective direction.
When the solution approximates the optimal, the second term would be smaller than the former term when the variances are large, then the value of the composite loss will increase, leading to the non-convergence.
This result is aligned with the non-convergence for MOGA algorithms.
We next give a second-order bias analysis of the composite gradient d k by the following lemma.
Lemma 8.
Under Assumption 1, 2, 3, for any gradient descent algorithm updated with composite gradient G k λ k , we have the following inequality
[Equation]
≤ mBm (cid:88) i =
V ξ k ( g i )
[Equation]
(cid:118)(cid:117)(cid:117)(cid:116)
V ξ k
[ λ k ] m (cid:88) i =
V ξ k
[ g i ( x k )] .
Proof.
We first decompose the expectation into
E ξ k
[Equation]
 (cid:124) (cid:123)(cid:122) (cid:125)
[Equation]
B .
We next analyze the term A. By the triangle inequality of the l 2 norm, we have term A =
E ξ k
[Equation]
(cid:88)
[Equation]

 =
E ξ k  
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) m (cid:88) i
= λ
ik ( ∇ f i ( x k ) − g i ( x k ))
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  
 ≤ E ξ k   (cid:32) m (cid:88) i = λ
ik (cid:13)(cid:13) ( ∇ f i ( x k ) − g i ( x k ))
(cid:13)(cid:13)(cid:33) 2   ,
Further, by the fact that λ ik ∈ [0 , B ] , i = 1 , . . .
, m , we know that term
A ≤ E ξ k (cid:34)(cid:32) m (cid:88) i
= λ i k(cid:33) (cid:32) m (cid:88)
i = (cid:13) (cid:13) ∇ f i ( x k )
− g i ( x k ) (cid:13) (cid:13) 2(cid:33)(cid:35) ≤ E ξ k (cid:34) mB 2 m (cid:88) i =
(cid:13)(cid:13) ( ∇ f i ( x k )
− g i ( x k ))
(cid:13)(cid:13)(cid:35) =
mBm (cid:88) i =
V ξ k ( g i ) .
We then analyze the term B. Similar with the proof in Lemma 7, and by the fact that the minus sign does not affect the inequality, we know that
[Equation]
Combining all the results, we obtain
[Equation]
Remark.
In the single-objective stochastic optimization, the second term is zero, leaving only the gradient variance in the bound.
While in the multi-objective stochastic optimization, the second term is non-zero and related to the number of objectives, the variance of composite weight, and the gradient variance, which makes it significantly different.
E.4 Proof for Theorem 2 (Convex Convergence)
Theorem 2.
Under Assumption 1, 2, 3.
For the sequence x 0 , x 1 , . . .
, x n generated by Algorithm 1, we assume objective functions f 1 ( x ) , . . .
, f m ( x ) are all convex with bounded optimal values as f 1 ( x ∗ ) , . . .
, f m ( x ∗ ) ≤ F for x ∗ in Pareto set, and the distance from sequence to Pareto set is bounded, i.e., ∥ x k − x ∗ ∥ ≤ D .
Set 0 ≤ η n ≤ . . .
≤ η 1 ≤ 1 /mLB, α k ∈ (0 , 1] for k = 1 , . .
. , n in
Algorithm 1, it achieves
[Equation]
To prove Theorem
[Equation]
it is sufficient to prove the bound for
[Equation]
We next provide the reason.
Specifically, from Assumption
[Equation]
we have (cid:80) mi =1 λ ik ≥ 1 for k = 1 , . . . , n .
Therefore
[Equation]
we know that
[Equation]
∗ k ∈K m (cid:88) i =
[Equation]
λ ik f i ( x ∗ k )] .
Furthermore, denote ¯ λ k = λ k / (cid:80) mj =1 λ jk .
By the property for the max-min operator, the left side of the above inequality can upper bound what we need to bound in Theorem 2 as
[Equation]
Therefore, the rest of the proof is to bound 6.
Before the final proof, we first introduce several
Lemmas.
The following Lemma give an induction as
Lemma 9.
Under the same assumption as Theorem 2, for any gradient descent algorithm updated with composite gradient G k λ k , and any x ∗ in Pareto set, we have the following inequality
[Equation]
Proof.
From the L -smoothness of each objective function, we have
[Equation]
By the iterative rule, we know that x k +
=
x k + η k d k .
Further, by the convexity of f i , and denote x ∗ as any point in the feasible set, we have f i ( x k ) ≤
f i ( x ∗ ) + ∇ f i ( x k ) ⊤ ( x k − x ∗ ) .
Then we can get
[Equation]
Summing up above for i = 1 , . .
, m , we have m
[Equation]
By setting η k ≤ 1 /mLB , we can know that Lη k (cid:0)(cid:80) mi =
λ ik (cid:1) ≤
Lη k mB ≤ 1 .
Then, the third term can be bounded as Lη 2 k 2 (cid:0)(cid:80) mi
=
λ ik
[Equation]
∥ d k ∥ 22 ≤ η k 2 ∥
d k ∥ 22 .
Plugging such a bound for the third term, and take the expectation on random variable ξ k , we then obtain
E
[Equation]
By the linearity of expectation, we rearrange the above term into
E ξ k [ m (cid:88)
[Equation]
We next to bound the first term.
By rearranging the equation, we get
E ξ k 
[Equation]

 =
[Equation]
where the last equation is by the fact that x k +
= x k + η k d k .
Since we know that
E ξ k
[Equation]
The second inequality is by the assumption that the set { x k , k = 1 , . .
, K } is a subset of a bounded set with diagram D , and the third one is from Lemma 1.
By combining all the above results, we have
E ξ k
[Equation]
[Table]
[Equation]
In addition, there is a slight different with the Lemma that we aim to prove: the left side is x k +1 not x k .
The following we fix this issue.
By the H-Lipschitz continuous for each objective, we know that
E ξ k
[ m
[Equation]
By the triangle inequality of norm and the definition for the composite gradient d k , we have
[Equation]
The third inequality is from the H-Lipschitz, and the last one is from Assumption 1.
By the fact that ab ≤ 12 β k a 2 + β k 2 b 2 for any β k > 0 , we can have
[Equation]
mBH.
Let β k = B √ m/ (cid:112)(cid:80) mi
=1 V [ g i ( x k ] , we have E ξ k
[ ∥ d k ∥ ] ≤ B (cid:112) m (cid:80) mi =1 V [ g i ( x k ] + mBH .
Plug in the result, we can get
[Equation]
Combining all the above, we prove the lemma.
By summing up the above lemma, we get the following one
Lemma 10.
Under the same assumption as Theorem 2, for any x ∗ in Pareto set, we have the following inequality for CRMOGA algorithm
[Equation]
Proof of Theorem 2.
From Lemma 2, we know that V ξ k
[ λ k ] ≤ m 2 B 2 (1 − α k ) 2 .
From Assumption 2, we know that V ξ k ( g i ( x k )) ≤ σ 2 .
Then plug into Lemma 9, we get m (cid:88)
[Equation]
∗ )]
≤ 1 2 η k E ξ k (cid:2)
∥ x k − x ∗ ∥ 22 − ∥ x k +1 − x ∗ ∥ 22 (cid:3) + mDBσ √ m (1 − α k ) + 6 m 2 B 2 Hσ √ m (1 − α k )
η k + m 2 B 2 ( σ 2 + σH + H 2 )
η k .
Sum up the above inequality from k = 1 to k = n , we have
[Equation]
Take expectation of ξ 1 , ξ 2 , . .
. , ξ n on the both sides, we can get n
[Equation]
By the fact that η k ≥ η k +
[Equation]
we then bound the first term as n
[Equation]
=
[Equation]
Plug in the above result
[Equation]
we have
[Equation]
We thus prove the lemma.
By observing Lemma 10 and the final version we need to proof
[Equation]
the only different is the compared optimum is x ∗ for all k not
the adaptive one x ∗ k = arg
min x ∈K (cid:80) mi =1 λ ik f i ( x ) , the following lemma measures such a gap.
Lemma 11.
Under the same assumption as Theorem 2, for any x ∗ in Pareto set
[Equation]
we have the following inequality for CRMOGA algorithm
[Equation]
(1 − α k ) m (cid:88) i = | λ il − ˆ λ il +1 |
Proof.
Denote ¯ λ i = (cid:80) nk
=1 λ ik /n .
Denote x ∗ =
arg min x ∗ ∈K (cid:80) nk
=1 (cid:80) mi =
λ ik f i ( x ∗ ) and x ∗ k = arg
min x ∗ k
∈K (cid:80) mi
=1 λ ik f i ( x ∗ k ) , k = 1 , . . . , n .
We first rearrange into
E (cid:34) n (cid:88) k
= m (cid:88) i = λ
ik f i ( x ∗ ) − n (cid:88) k
= m (cid:88) i = λ
ik f i ( x ∗ k ) (cid:35) =
E (cid:34) n (cid:88) k
= m (cid:88) i =
¯ λ i f i ( x ∗ ) − n (cid:88) k
= m (cid:88) i = λ
ik f i ( x ∗ k ) (cid:35) .
Since x ∗ is the minimum for ¯ λ ik f i ( x ) , we then have
E (cid:34) n (cid:88) k
= m (cid:88) i = λ
ik f i ( x ∗ ) − n (cid:88) k
= m (cid:88) i = λ
ik f i ( x ∗ k ) (cid:35) ≤ E (cid:34) n (cid:88) k
= m (cid:88) i =
¯ λ i f i ( x ∗ k ) − n (cid:88) k
= m (cid:88) i = λ
ik f i ( x ∗ k ) (cid:35) =
E (cid:34) n (cid:88) k
= m (cid:88)
i =
(¯ λ i − λ ik )
f i ( x
[Equation]
By the assumption that objective functions f 1 ( x k ) , . . .
, f m ( x k ) ≤ F are all bounded for k = 1 , . . .
, n , we get
E
[Equation]
As we know that n (cid:88) k
= m
[Equation]
i = |
[Equation]
= n (cid:88) k
= m (cid:88) i
= |n n (cid:88) t
= ( λ it − λ ik ) | = 1 n n (cid:88) k
= m (cid:88) i = | n (cid:88) t
= ( λ it − λ ik )
| ≤n m (cid:88) i = n (cid:88) k
= n (cid:88) t =
| λ it − λ ik | = 1 n m (cid:88)
i = n (cid:88) k
= k − 1 (cid:88) t
= | λ it − λ
ik | + 1 n m (cid:88) i = n (cid:88) k
= n (cid:88) t = k
+ | λ it − λ
ik
| ≤n m (cid:88) i = n (cid:88) k
= n (cid:88) t =
k t (cid:88) l
=
k | λ il − λ il +1 | = 2 n m (cid:88)
i = n (cid:88) k
= n (cid:88) l = k n (cid:88) t
= l | λ il − λ il +1 | =n m (cid:88) i = n (cid:88) k
= n (cid:88) l = k ( n − l + 1)
| λ i l − λ i l +1 | ≤m (cid:88)
i = n (cid:88) k
= n (cid:88) l = k | λ i l − λ i l +1 | =m (cid:88) i = n (cid:88)
l = l (cid:88) k
= | λ il − λ il +1 | =m (cid:88) i = n (cid:88)
l
= l | λ il − λ il +1 | =m (cid:88) i = n (cid:88)
l = l (1 − α k )
| λ il − ˆ λ il +1 | =n (cid:88) l = l (1 − α l +1 ) m (cid:88) i = | λ il − ˆ λ il +1 | .
The last two equation is by the definition of Algorithm
We are now ready to proof Theorem 2.
Combine Lemma 10 and Lemma 11, we have n (cid:88) k
=
E [ m (cid:88) i =
[Equation]
m (cid:88)
[Equation]
(cid:19)
[Equation] [Equation]
(cid:88) k
= (1 − α k ) + 6 m 5 / 2 B 2 Hσ n (cid:88) k
= (1 − α k ) η k + m 2 B 2 ( σ 2 + σH + H 2 ) n (cid:88) k
= η k + 2 F n (cid:88) l =
[Equation]
[Equation] [Equation]
(cid:88)
[Equation]
|
[Equation]
| .
By averaging the above, and the fact that η n ≤ η 1 , we prove the theorem.
E.5 Proof for Theorem 3 (Non-convex Convergence)
Theorem 3.
Under Assumption 1, 2, 3.
For the sequence x 0 , x 1 , . . .
, x n generated by Algorithm 1, we assume objective functions
f 1 ( x k ) , . . .
, f m ( x k ) ≤ F are all bounded for k = 1 , . . . , n .
Set 0 ≤ η n ≤ . . .
≤ η 1 ≤ 1 /mLB and α k ∈ (0 , 1] in Algorithm 1, it achieves
[Equation]
Similar to the convex case, to prove Theorem 3, it is sufficient to prove the bound for
[Equation]
where λ k is generated by MOGA or CRMOGA algorithms.
The reason is also similar to the convex case, which is that 7 upper bounds the left side in Theorem 3 because
[Equation]
where ¯ λ k = λ k / ( (cid:80) mi =1 λ ik ) 2 ∈ S m .
The first inequality of the above is due to Assumption 1 that (cid:80) mi
=1 λ ik ≥ 1 .
We next present necessary lemmas.
Lemma 12.
Under the same assumption as Theorem 3, for any gradient descent algorithm updated with composite gradient G k λ k , we have the following inequality
[Equation]
Proof.
From the L -smoothness of each objective function, we have
[Equation]
Sum up both side for i = 1 , . . .
, m , and take the expectation on random variable ξ k , we can get
E ξ k [ m (cid:88) i =
[Equation]
The last inequality is by the Assumption that λ ik ≤ B , then E ξ k (cid:2)(cid:80) mi =1 λ ik ∥ d k ∥ 22 (cid:3) ≤
E ξ k (cid:2)(cid:80) mi =1 B ∥ d k ∥ 22 (cid:3) =
mB
E ξ k (cid:2) ∥
d k ∥ 22 (cid:3) .
From the result of Lemma 7, we can bound the first term and obtain
E ξ k [ m (cid:88) i = λ
ik f i ( x k +1 ) − m (cid:88) i = λ
ik f i ( x k )] ≤ 2 η k mBH (cid:118)(cid:117)(cid:117)(cid:116)
[Equation] [Equation]
Then, adopting the result from Lemma 8, we know that
E ξ k [ m (cid:88) i =
[Equation]
By the fact that η k ≤ 1 /LmB , we further have
E ξ k
[ m
[Equation]
=
[Equation]
[Equation] [Equation]
[Table]

[Equation]
m (cid:88)
[Table]
=
[Table]
V
[Equation]
g
[Equation] [Equation]
By rearrangement, we therefore have η k
[Equation]
k 
[Equation] [Equation]
Take expectation of ξ 1 , ξ 2 , . .
. , ξ n on the both sides, we finally get η k
[Equation]
 
[Equation]
m
[Equation]
V
[Equation]
g i )
[Equation]
Lemma 13.
For CRMOGA algorithm, and under the same assumption with Theorem 3, we have n (cid:88) k
=  η k E [ m (cid:88) i =
[Equation]
Proof.
First, we can decompose the left side as n (cid:88) k
=  η k E [ m (cid:88)
i = λ
ik f i ( x k ) − m (cid:88) i = λ
ik f i ( x k +1 )]
= n (cid:88) k
=
E [ 1 η k m (cid:88) i = λ
ik f i ( x k ) − η k −m (cid:88) i = λ
ik − 1 f i ( x k
[Equation]
+ E [ 1 ηm (cid:88)
[Equation]
By the definition of CRMOGA algorithm, we know that λ
ik = α k λ ik − 1 + (1 − α k )ˆ λ ik , i = 1 , . . .
, m , where { ˆ λ ik } mi =1 are calculated by MOGA algorithms.
Then we have the following decomposition for the first term n (cid:88)
[Equation]
Combining the above, and by the assumption that | f i ( x k ) | ≤ F, λ ik , ˆ λ ik ∈ [0 , B ] as well as the nonincreasing for η k , we therefore have
[Equation]
Now, we can present the final proof.
From Lemma 2, we know that V ξ k
[ λ k ] ≤ m 2 B 2 (1 − α k ) 2 .
From Assumption 2, we know that V ξ k ( g i ( x k )) ≤ σ 2 .
Then plug into Lemma 12, we get
E
[Equation]
m (cid:88)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
The above is divided with η k / 2 for the both side.
Then, sum up the above inequality from k = 1 to k = n and take the result from Lemma 13, we have n (cid:88) k
=
[Equation]
E
[Equation]
(cid:88)
[Equation] [Equation]
≤
[Equation]
≤ F n (cid:88)
[Equation] [Equation]
k = η k .
By averaging the above inequality, we prove the theorem.
E.6 Proof of Corollary
Corollary 1.
(a) Set 1 /mLB ≥ η k
= O (1 / √ k ) , α k = max { 1 −
η k /η 1 , 1 − η k / ( η 1 k (cid:80) mi =1 | λ ik − 1 − ˆ λ ik | ) } in Theorem 2, Algorithm 1 converges with O ( n 1 / 2 ) rates in the convex case.
(b) Set 1 /mLB ≥ η k
= O (1 / √ k ) , α k = max { 1 −
η k /η 1 , 1 − η k / ( η 1 √ k (cid:80) mi =1 | λ ik − 1 − ˆ λ ik | ) } in Theorem 3, Algorithm
O ( n 1 / 4 ) rates in the non-convex case.
Proof.
For the convex case (a) , α k = max { 1 − η k /η 1 , 1 − η k / ( η 1 k (cid:80) i ∈ [ m ] | λ ik − 1 − ˆ λ ik | ) } means 1 − α k = min {
η k /η 1 , η k / ( η 1 k (cid:80) i ∈ [ m ] | λ ik − 1 − ˆ λ ik | ) } .
Plug in Theorem 2, we have
[Equation]
For the non-convex case (b) , α k = max { 1 − η k /η 1 , 1 − η k / ( η
[Equation]
√ k max i ∈ [ m ] | λ ik − 1 − ˆ λ ik | ) } means 1 − α
[Equation]
= min {
[Equation]
η
[Equation]
√ k
[Equation]
| λ ik − 1 − ˆ λ ik | ) } .
Plug in Theorem 2, we have
[Equation]
Remark 1.
Also, it is not difficult to verify that setting α k = max { 1 − η k /η 1 , 1 − η k / ( η 1 k (cid:80) i ∈
[ m ] | λ ik − 1 − ˆ λ ik | ) } also lead to the same convergence in the non-convex case.
For simplicity, we present the unified version in the main text, and provide the complete version in the
Appendix.
Remark 2.
Since multi-objective optimization could reduce to single-objective optimization if the multiple objectives are the same, the convergence rate of stochastic MOO will be no better than single-objective SGD.
Note that our convergence rates align with those in single-objective SGD [7, 2], which suggests that our bounds are tight in order.
In comparison with stochastic MOGA, we have demonstrated clearly in this paper that stochastic MOGA algorithms fail to converge to the
Pareto optimal/critical points in Section 4, so it would be less meaningful to compare them.
As for comparing with full-batch MOGA, the comparison is similar with GD and SGD.
E.7 Discussion on Theorem 2 and Theorem
Average scheme.
The averaging scheme for Theorem 2 and Theorem 3 can be transformed to traditional ones.
It just needs to use a typical uniform sampling technique in stochastic optimization [4, 26, 7, 22] to modify the algorithmic output.
Specifically, the output is changed from x n to ¯ x n that is uniformly sampled in { x 1 , x 2 , . . .
, x n } .
Then, by this randomness and the linearity of expectation, we can know that
[Equation]
and
[Equation]
Therefore, we recover the traditional convergence bound for ¯ x n .
Assumptions of bounded gradient.
The assumption of bounded gradients appears very widely in related literature on stochastic optimization.
Specifically, in the convex setting, the bounded gradient assumption is used to derive the convergence bound for various famous optimizers, such as Adam (Theorem 4.1 in [24]), Adagrad (Theorem 5 in [8]), AMSGrad (Theorem 4 in [38]), etc.
In the non-convex setting, it helps to analyze the stochastic convergence of momentum (Theorem 1 &in [46], Assumption 3 in [4], Assumption 2 in [47], and Assumption 1(ii) in [23]).
Additionally, in practice with deep learning, there are many techniques that are designed to avoid gradient explosion, such as weight regularization [37] and gradient clipping [49].
These techniques ensure the boundness of the gradient in practice, which validates this assumption.
Furthermore, our theoretical results only require that ∥∇ f i ( x k ) ∥ ≤ H for i = 1 , . . .
, m and k = 1 , . .
. , n , i.e., the gradients along the optimization path are bounded, which is also supported by our experiments.
Assumptions of bounded function.
The assumption of bounded functions has been used in recent studies on non-convex stochastic optimization, such as analyzing SGD (Assumption 2 in [9]), and analyzing SGD+momentum (Assumption in Section 3 of [27]).
Actually, this assumption can be further relaxed in our analysis.
Specifically, in the convex setting, it can be relaxed to that the function values of Pareto optimal solutions are bounded, i.e., | f i ( x ∗ ) | ≤ F, i = 1 , . . .
, m , if x ∗ is Pareto optimal.
The reason is that we only require this condition in Lemma 11.
The relaxed assumption is surely satisfied since the optimal solutions naturally have bounded function values.
In the non-convex setting, note that in the related papers analyzing SGD (Remark 2 in the analysis of SGD [7]) as well as MOGA (Theorem 3.1 in the analysis of MGDA [11]) with non-convex functions, it is commonly assumed that the initial function values
f i ( x 1 ) , i = 1 , . . .
, m are bounded.
Essentially, our analysis only requires that f i ( x k ) , i = 1 , . . .
, m are bounded for k = 1 , . . . , n , since we only use it in Lemma 13.
This assumption is mild in practice as long as the function values of iterates along the practical training path are bounded.
In fact, all our experiments indicate that the function values are well located in a bounded regime.
F Convergence Analysis for Strongly Convex Functions
Theorem 6.
Under Assumption 1, 2, 3.
For the sequence x 0 , x 1 , . . .
, x n generated by Algorithm 1, we assume objective functions f 1 ( x ) , . . .
f m ( x ) are all µ -strongly convex and bounded as f 1 ( x ) , . .
f m ( x ) ≤ F , and the distance from sequence to Pareto set is bounded, i.e., ∥ x k − x ∗ ∥ ≤ D for x ∗ in Pareto set.
Set 0 ≤ η n ≤ . . .
≤ η 1 ≤ 1 /mLB, α k ∈ (0 , 1] for k = 1 , . .
. , n in Algorithm 1, it achieves
E (cid:34) m
[Equation]
i
[Equation]
−
[Equation]
) (cid:35)
[Equation]
(cid:34) m
[Equation]
i = λ
[Equation]
f
[Equation]
Proof.
By the property of µ -strongly convex ((4.12) from [2]), we have
[Equation]
where x ∗ k = arg
min x (cid:80) mi =
λ ik f i ( x ) .
Recall Lemma 12, we know that
[Equation]
By combining the above
[Equation]
we get
[Equation]
By rearranging the inequality, we further have
E
[Equation]
i =
[Equation]
By the optimality of x ∗ k , we know that (cid:80) mi =1 λ ik f i ( x ∗ k ) ≤ (cid:80) mi =1 λ ik f i ( x ∗ k +1 ) .
Therefore, we can get
E ξ k (cid:34) m (cid:88) i =
[Equation]
g
[Equation]
Further by decomposing, we can obtain m (cid:88) i = λ
[Equation]
− m (cid:88) i = λ
ik +
f i ( x ∗ k +1 )
= m (cid:88) i = λ
ik f i ( x k +1 ) − m (cid:88) i = λ ik f i ( x ∗ k +1 ) − m (cid:88) i =
( λ ik − λ ik +1 )
f i ( x k +1 )
+ m (cid:88) i
=
( λ ik − λ ik +1 )
f i ( x ∗ k +1 ) .
By the definition of Algorithm 1 and the boundness of gradient function.
We know that λ
ik +1 − λ ik =
(1 − α k +1 )
( ˆ λ ik +1 − λ ik ) and | f i ( x k ) | ≤ F, | f i ( x ∗ k ) | ≤ F.
Then, we can easily get that − ( λ ik − λ ik +1 )
f i ( x k +1 ) ≤ F (1 − α k +1 ) | ˆ λ ik +1 − λ ik | and ( λ ik − λ ik +1 ) f i ( x ∗ k +1 ) ≤ F (1 − α k +1 ) | ˆ λ ik +1 − λ ik | .
Thus, we have m (cid:88) i = λ
ik +1 f i ( x k +1 ) − m (cid:88) i = λ
ik +
f i ( x ∗ k +1 ) ≤ m (cid:88) i = λ
ik f i ( x k +1 ) − m (cid:88) i = λ
ik f i ( x ∗ k +1 ) + 2 F m (cid:88) i =
(1 − α k +1 ) | ˆ λ ik +1 − λ ik | .
Then combining the above, we now obtain
E ξ k (cid:34) m (cid:88) i =
λ
i k
+
f i ( x k +1 ) − m (cid:88) i
= λ
i k
+
f i ( x ∗ k +1 )
(cid:35) ≤ (1 − η k µ ) E ξ k (cid:34)
[Equation]
From Lemma 2, we know that V ξ k
[ λ k ] ≤ m 2 B 2 (1 − α k ) 2 .
From Assumption 2, we know that
V ξ k ( g i ( x k )) ≤ mσ 2 .
Then plug into the above, we get
E
[Equation]
k
[Equation]
m
[Equation]
Taking total expectations, this yields
[Equation]
We now are able to provide the convergence analysis for Algorithm 1 with fixed stepsizes.
Corollary 2.
Set 0 < η k = η ≤ min { 1 mLB , 1 L } and α k = max { 1 − η, 1 − η 2 / ( (cid:80) mi =1 | λ ik − 1 − ˆ λ ik | ) } in Theorem 6.
Algorithm 1 linearly converges to a solution near the Pareto optimal, and the expected gap limits to ηM/µ , where constant M = 4 m 5 / 2 B 2 H
+ Lm 3 B 2 σ 2+ 2 F .
Proof.
From the parameter setting, we know that 1 − α k = min { η, η 2 / ( (cid:80) mi =1 | λ ik − 1 − ˆ λ ik | ) } , which suggests that 1 − α k ≤ η and (1 −
α k +1 )
(cid:80) mi =1 | ˆ
λ ik +1 − λ ik | ≤ η 2 .
By Theorem 6, we have
E
[Equation]
i
[Equation]
i
[Equation] [Equation]
η 2 M, where M = 4 m 5 / 2 B 2 H
+
Lm 3 B 2 σ 2+ 2 F .
Then, subtracting the constant ηM/µ from both sides, we obtain
[Equation]
Observe that the above is a contraction inequality, by the fact that
[Equation]
The second inequality is by the range of η .
The result thus follows by applying the above contraction inequality repeatedly through iteration k = 1 , . .
. , n .
[Equation]
If n → ∞ , then the suboptimal gap for the solution will limit to ηMµ .
Remark.
Here, the algorithm does not converge to Pareto optimal but is a suboptimal solution near an optimal solution.
This result aligns with the one in single-objective stochastic optimization, and thus is tight in order.
Note that this gap is unavoidable, as is the case in single-objective SGD, because the stepsize needs to tradeoff the stochastic variance and the convergence rate (the discussion of Theorem 4.6 in [2]).
Additionally, the suboptimal gap is related to the scale of the stepsize η , and hence we can set sufficiently small η to get better convergence.
To guarantee the convergence to Pareto optimal solutions, we can use a diminishing stepsize as the case in single-objective stochastic optimization [2].
The result is as follows.
Corollary 3.
Set η k = βk + γ , β > 1 µ where γ > 0 such that η k ≤ 1 mLB , and α k = max { 1 − η k /η 1 , 1 − η 2 k / ( η 1 (cid:80) mi =1 | λ ik − 1 − ˆ λ ik | ) } in Theorem 6.
Algorithm 1 linearly converges to Pareto optimal with O (1 /n ) rates.
Proof.
Specifically, we prove the following result for k = 1 , . . . , n
[Equation]
m (cid:88) i
= λ
ik f i ( x k ) − m (cid:88) i = λ
ik f i ( x ∗ k ) (cid:35) ≤ v k + γ , where v satisfies that v = max { β 2 M βµ − 1 , (1 + γ )( (cid:80) mi =1 λ i 1 f i ( x 1 ) − (cid:80) mi
=1 λ
i 1 f i ( x ∗ 1 )) } .
From Theorem 6, we know that
E (cid:34) m (cid:88) i = λ
ik +1 f i ( x k +1 ) − m (cid:88) i = λ
ik +
f i ( x ∗ k +1 )
(cid:35) ≤ (1 − η k µ ) E (cid:34) m (cid:88) i = λ
ik f i ( x k ) − m (cid:88) i = λ
ik f i ( x ∗ k ) (cid:35) + 4 m 5 / 2 B 2 Hη k (1 − α k ) + Lm 3 B 3 σ 2 η 2 k+ 2 F (1 − α k +1 ) m (cid:88) i = | ˆ λ ik +1 − λ ik | ≤ (1 − η k µ ) E (cid:34) m (cid:88) i = λ
ik f i ( x k ) − m (cid:88) i = λ
ik f i ( x ∗ k ) (cid:35) + η 2 k ( 4 m 5 / 2 B 2 H η+
Lm 3 B 2 σ + 2 F η 1 ) +
Fη 2 k + η≤ (1 − η k µ ) E (cid:34) m (cid:88)
i = λ
ik f i ( x k ) − m (cid:88) i = λ
ik f i ( x ∗ k ) (cid:35) + η 2 k (4 m 5 /
[Equation]
where the constant in the above is defined as M = 4 m 5 / 2 B 2 H η+
[Equation]
B
[Equation]
+ 4 Fη 1 for simplicity.
The last inequality is by the fact that η k ≥ η k +1 .
We then prove the corollary by mathematical induction.
First, the parameter setting here ensures the result holds for k = 1 .
Next, we assume the inequality of the theorem holds for k , which holds
[Equation]
Then by the parameter setting, for the case of k + 1 , we have
[Equation]
where the last inequality is by the condition of v .
By the fact that ( a − 1)( a + 1) ≤ a 2 for a > 0 , we have k + γ − 1 ( k + γ
[Equation]
2 ≤
[Equation]
Plug in the above, we can prove the result for k + 1 .
Thus, we theorem holds for k = 1 , . . . , n , which suggests an O (1 /n ) convergence.
Remark 1.
The result from Corollary 3 shows the convergence rate is aligned with single-objective
SGD with adaptive stepsize, suggesting that Algorithm 1 is tight in order with strongly convex functions.
Remark 2.
The bounded function assumption in Theorem 6 can be relaxed to the combination of the assumptions in the convex and non-convex settings.
Specifically, the function values of Pareto optimal solutions are bounded, i.e., | f i ( x ∗ ) | ≤ F, i = 1 , . . .
, m , and the function values of iterates along the practical training path are bounded, i.e., f 1 ( x k ) , . .
f m ( x k ) ≤ F for k = 1 , . .
n.
G Additional Experiments and More Details G.1 More Insights of CR-MOGA
How the non-convergence reflects in practice.
In fact, the similar situation reflected by the counterexample is not rare in practice, especially when the stochastic noises are large.
For example, we have added an additional experiment of MultiMNIST by setting batch size from 8 to 2, representing increasing levels of stochastic noises (a smaller batch size introduces a larger stochastic noise in the mini-batch gradient).
We report the training losses for the final outputs of MGDA and CR-MGDA as follows, representing the comparison of MOGA and CR-MOGA.
From results of Table 2, we observe that the loss of vanilla MGDA becomes significantly larger when the batch size is smaller, while CR-MGDA appears not sensitive to the batch size.
This possible reason is that the convergence issue of MOGA becomes more severe with larger stochastic noise, which severely degrades the performance of MOGA.
Since CR-MOGA is proven to converge to the Pareto optimal/critical, it is much more robust to stochastic noises.
Thus, in this regard, our method is more practical for real-world applications.
Comparison with the full-batch MOGA.
Although there are no guarantees for the closeness between λ calculated by CR-MOGA and the actual λ calculated by MOGA with full-batch gradients.
As shown in Figure 4, we empirically observe that the λ k of CR-MOGA is very close to the actual λ ∗ k yielded by the MOGA with full batch gradients.
Specifically, the gaps between λ k of CR-MOGA and actual λ ∗ k are below 0.01 with low error bars in the toy example, while the gaps of vanilla MOGA algorithms are all averagely larger than 0.10 with large error bars.
[Figure] [Figure]
G.2 Additional Stochastic Convex Simulation
In this simulation
[Figure]
we establish a similar setting with Section
[Figure]
where we aim to optimize x ∈ Rto attain smaller distances to two fixed points a = [ − 3 , 0] , b
= [3 , 0] .
The first objective is f 1 ( x ) =
∥ x − a ∥
[Figure]
and the second one is f 2 ( x ) =
∥ x − b ∥ 22 .
The stochastic gradient for each objective can be represented as g 1 ( x ) = x − a + ξ 1 and g 2 ( x ) = x − b + ξ 2 .
The stochastic noises ξ 1 , ξ 2 are two dimensional random noise with zero-mean Gaussian distribution, where the covariance matrix for ξ 1 is [[3 , 2] , [2 , 3]] and for ξ 2 is [[3 , − 2] , [ − 2 , 3]] .
Figure 5 demonstrates the distance to Pareto set.
We observe that all three MOGA algorithms can not converge to the Pareto optimal points, while all CRMOGA algorithms can approach the Pareto set.
This result is aligned with the result in Section 4, and the more general Gaussian noise shows that the non-convergence is not special in stochastic multiobjective optimization.
G.3 Implementation Details of Experiments
The implementation for CRMOGA is very simple, just by adding one additional line that exponentially smoothes the composite weights to MOGA algorithms.
Note that this mechanism does not require additional computational cost and is flexible to be implemented in MOGA algorithms.