Multi-Objective Deep Learning with Adaptive Reference Vectors
Abstract
Many deep learning models involve optimizing multiple objectives.
Since objectives are often conﬂicting, we aim to get diverse and representative trade-offsolutions among these objectives.
Gradient-based multi-objective optimization(MOO) algorithms using reference vectors has shown promising performance.
However, they may still produce undesirable solutions due to mismatch betweenthe pre-speciﬁed reference vectors and the problem’s underlying Pareto front.
Inthis paper, we propose a novel gradient-based MOO algorithm with adaptive reference vectors.
We formulate reference vector adaption as a bilevel optimizationproblem, and solve it with an efﬁcient solver.
Theoretical convergence analysis isalso provided.
Experiments on an extensive set of learning scenarios demonstratethe superiority of the proposed algorithm over the state-of-the-art.
Introduction
Deep learning models are often evaluated under multiple, potentially conﬂicting, criteria.
For example, 13 in multi-task learning [6], a single model is required to perform well on multiple tasks.
In somescenarios, besides accuracy, model fairness is also important so as to ensure that the model is not 15 biased against gender and race.
These problems can all be formulated as multi-objective optimization(MOO)
[36] problems, and have attracted attention from various ﬁelds such as energy resource 17 optimization [8] and signal processing [4].
Since the multiple objectives usually cannot be optimized simultaneously by a single solution, the 19 goal of MOO is to ﬁnd a set of solutions with different trade-offs to approximate the true Pareto front(PF).
The past decades have witnessed the birth of a large number of gradient-free MOO algorithms, 21 such as evolutionary multi-objective optimization algorithms [10, 49] and Bayesian multi-objectivealgorithms [23, 3].
These algorithms perform well on small-scale problems but fail to provide useful 23 solutions when facing the huge number of parameters in deep learning models.
Recently, gradient-based MOO algorithms [13] have demonstrated promising performance in deep 25 learning.
Based on the pioneering work in [42], Lin et al.
[28] and Mahapatra et al.
[34] propose togenerate multiple solutions using a set of reference vectors (RV).
These algorithms try to search for 27 solutions on the PF closest to each RV.
Some strategies are further proposed to improve the efﬁciencyby training a hypernetwork [37, 27] or reference-vector-conditioned network [41].
However, note 29 that a set of uniformly distributed RVs may not lead to a set of uniformly distributed solutions.
Hence,an important limitation of these algorithms is that solutions generated ﬁxed RVs may not cover some parts of the to provide enough information about the PF an illustration, in the obtained solution the
(Figure 1b)
[Figure]
Since it is impossible to know the true PF of the problem before optimization, how to generate a proper RV set is the main challenge.
In this paper, we propose to learn the set of RVs simultaneously with the model parameters.
This is formulated as a bilevel optimization problem [12], in which the lower-level optimization problem obtains the Pareto-optimal solutions with the given set of RVs, while the upper-level problem optimizes the RVs based on some quality measure.
Moreover, we propose an inexpensive solver for this optimiza- tion problem, while still showing nice theoretical convergence properties.
Experiments on an ex- tensive set of learning scenarios demonstrate the superiority of the proposed
Related Work: Multi-Objective Optimization (MOO)
In multi-objective optimization (MOO) [36], one aims to minimize m  2 objectives{ f 1 (  ) , f 2 (  ) , . . .
f m (  ) } , or, equivalently, the vector-valued function:
[Equation]
A solution  1 is dominated by another solution  2 if and only if f i (  1 )  f i [Equation]  2 ) for i 2 [ m ] ⌘{ 1 , . . .
, m } , and 9 i 2 [ m ] , f i (  1 )
f i (  2 ) .
A solution  ⇤ is Pareto-optimal if and only if it is not 39 dominated by any other  0 .
A Pareto front (PF) is the set of multi-objective values of all Paretooptimal solutions.
A PF is regular [45] if its shape is simplex-like (i.e., all vectors with positive 41 directions intersect it when they start at the origin), and irregular otherwise.
It is shown that regular
PFs are not very realistic [22].
Since the number of Pareto-optimal solutions is usually large or even inﬁnite, a set of n Pareto- 44 optimal solutions  = {  1 , . .
,  n } is often used to approximate the PF.
Denote the corresponding 45 multi-objective values as F = { f (  1 ) , . . .
, f (  n ) } .
The quality of  can be evaluated from two 46 perspectives: convergence and diversity [24].
Convergence refers to the distance between F and the 47 true PF, while diversity measures whether the solutions are well-distributed in the space of objectives.
A popular measure, which evaluates both convergence and diversity, is the hypervolume (HV) [50].
Given a reference point z 2 R m , the HV of F is:
[Equation]
where [ z, f i ] ⌘ { q 2 R m | f i  q  z } , and  ( · ) is the Lebesgue measure of a set.
A reference vector (RV) [49], sometimes called the weight vector or preference vector, can be used to 52 guide the optimization algorithm by indicating the preferred point on the PF.
Usually, an algorithm isexpected to obtain the Pareto-optimal solution closest to the given RV in the objective space.
Given 54 a RV r 2 R m + , the MOO problem in (1) can be converted to a single-objective problem by using a 55 scalarization function s (  ; r ) .
The most straightforward construction is linear scalarization:
[Equation]
which weights the m objectives with the elements r ( j ) ’s of r .
It is known that the minimizer of 57 s (  ; r ) is also Pareto-optimal for the original problem in (1) (Ch 4.7, [5]).
However, the solutionobtained may be far way from the given RV.
To encourage the solutions to be closer to this vector, a 59 penalty term can be added to the linear scalarization function [41]:
[Equation]
where  is a constant.
Given a (discrete) set of solutions, Ma et al.
[33] generates exploration 61 directions to spawn new solutions on the PF, leading to a continuous PF.
By using n RVs from a subset R ✓ R m ⇥ n + , a set of solutions can be obtained to approximate theentire PF.
Usually, R is simply set to R m ⇥ n + .
However, sometimes the decision-makers may only be 64 interested in a speciﬁc region of the PF [2].
For example, we may limit the angle between any RVand each coordinate axis.
R can then be changed to:
[Equation]
where u i is the i th coordinate axis and ' 1 , ' 2 are the maximum and minimum allowable angles, 67 respectively.
However, note that the set of pre-speciﬁed RVs may not ﬁt the problem’s PF (e.g., someof them may not intersect the underlying PF), leading to an undesirable solution distribution.
Gradient-Free MOO.
Evolutionary MOO algorithms (e.g., NSGA-II[10], MOEA/D[49]) and 70 Bayesian MOO algorithms (e.g., BMOA[23], USeMO[3]) are widely used for small-scale black-boxproblems.
These algorithms assume that gradient information is not available.
Hence, they often fail 72 to converge on deep learning problems where the solution space can be very large.
Gradient-Based MOO.
Gradient-based MOO algorithms are more efﬁcient when problems have 74 differentiable objectives and a large number of parameters.
Sener and Koltun [42] propose to apply
Multiple-Gradient Descent Algorithm (MGDA) [13] to multi-task learning.
Liu and Vicente [30] 76 provide theoretical analysis of stochastic MGDA.
MGDA can be further extended to incorporate
RVs (e.g., EPO [34, 35], ParetoMTL [28]).
Some algorithms (e.g., MOO-SVGD [31], HIGA [46]) 78 can output a solution set without using RVs.
However, they optimize several neural networkssimultaneously, and so are computationally expensive and need large GPU memory (especially when 80 the neural network is large).
Proposed Algorithm
As mentioned in Section 1, the ﬁxed uniformly distributed RVs used in common practice may result 83 in undesirable solution distributions.
Instead of using a ﬁxed set of n RVs ( R =
[ r 1 , . .
, r n ] 2 R ), 84 we propose to adapt them so that the resultant solution set is well-distributed.
In Section 3.1, weﬁrst introduce a reference vector-conditioned neural network so that reference vectors can be easily 86 handled without using a lot more parameters.
Section 3.2 then formulates reference vector adaptationas a bilevel optimization problem, and an efﬁcient solver is proposed in Section 3.3.
Its convergence 88 properties are then studied in Section 3.
3.1 Reference Vector-Conditioned Neural Network
In deep learning models,  corresponds to the network parameters and optimizing  = {  1 , . . .
,  n } 91 means optimizing n neural networks (as in EPO [34] and MOO-SVGD [31]), which is highlyinefﬁcient.
To alleviate this problem in deep MOO algorithms, Navon et al.
[37] and Lin et al.
[27] 93 propose to train a single hypernetwork [20] that can output neural network parameters based onthe RV.
In particular, the Pareto Hypernetwork (PHN) in [37] proposes two ways to optimize the 95 hypernetwork, leading to (i) PHN-LS, which uses linear scalarization, and (ii) PHN-EPO, whichuses EPO.
However, the hypernetwork incurs signiﬁcant computational overhead.
For example, 97 the hypernetwork in [37] is around 100 times larger than the base neural network.
In this paper,we use the more efﬁcient conditioned network [14, 41].
Speciﬁcally, Ruchte and Grabocka [41] 99 concatenates the RV and data sample, and treat this as a joint input to the network.
On the otherhand, YOTO [14] is originally developed for use with a family of parameterized loss functions, in 101 which the loss parameter is incorporated with the sample into the network via FilM layers [39].
Itis theoretically shown that YOTO is as powerful as using n neural networks [14].
In this work, we 103 adapt the architecture in [14] by replacing the loss parameter with the RV.
In this way, the proposedmodel only has a small parameter overhead compared to the single deep network.
3.2 Reference Vector Adaption via Bilevel Optimization
Consider a RV-conditioned neural network f (  , r ) with parameter  and RV r as input.
Witha set of RVs R =
[ r 1 , . . . , r n ] , the multi-objective values of a solution set can be written as[ f (  , r 1 ) , . . .
, f (  , r n )] .
We use a function ˆ Q ( · ) to measure its quality.
Two choices are con- 109 sidered in this paper.
The ﬁrst one encourages the f (  , r i ) ’s are far away from each other (and thus more uniformly distributed in the space of objectives):
[Equation]
where h is a constant.
The second one is:
[Equation]
which encourages the maximization of HV in (7).
Note that the HV-optimal solution is usually not 113 uniformly distributed [19, 44].
To obtain the set of RVs R that generates  , we formulate it as a bilevel optimization problem [12].
Recently, bilevel optimization has gained great popularity in many machine learning problems suchas meta-learning [16], neural architecture search [29], and hyperparameter optimization [17].
We 117 consider the following bilevel optimization problem:
[Equation]
where Q ( R,  ⇤ ( R )) ⌘  ˆ Q ( R,  ⇤ ( R )) , S ( R,  ) ⌘ P ni
[Equation] 1 s (  ; r i ) , and s (  ; r ) is the scalarization 119 function in (3) or (4).
The lower-level optimization problem (9) obtains the Pareto-optimal solutionswith the given set of RVs R , while the upper-level optimization problem (8) optimizes R to maximize 121 the corresponding solution quality.
The idea of RV adaption is also used in some evolutionary algorithms [40, 26].
However, they update 123 the RVs using information from the current population and archive, and cannot be directly used ingradient-based MOO algorithms.
Moreover, they do not have any theoretical guarantees.
3.3 Solving the Bilevel Optimization Problem
There are various bilevel optimization solvers for (8).
Many of them involve propagation through the 127 inner loop [16, 18], which has a large computational overhead compared to [41].
Here, we proposean inexpensive solver that still has theoretical guarantees (Section 3.4).
Essentially, it performs only 129 one stochastic gradient descent step in both the inner and outer loops.
The proposed procedure, shown in Algorithm 1, is called Gradient-based Multi-Objective 131 Optimization algorithm with Adaptive Reference vectors (GMOOAR).
In each iteration k , minibatches ⇠ k and ⇡ k are randomly sampled from the data and then used to estimate the stochastic 133 gradients r  S ( R k ,  k ; ⇠ k ) and r R Q ( R k ,  k +1 ; ⇡ k ) .
proj R ( · ) is the Euclidean projection operator 134 onto R that ensures that RVs are inside R .
In the sequel, the algorithm using uniformity-related 135 quality function (6) is denoted GMOOAR-U, while the one using (7) is denoted GMOOAR-HV.
The proposed algorithm has O ( w + m ) memory and O ( w + m ) time complexity per iteration, where 137 w is the dimension of  .
Since m ⌧ w in most cases, the proposed algorithm has comparable time 138 and space complexity with COSMOS, which is O ( w ) .
In this section, we provide convergence analysis for Algorithm 1.
As in [21], we make the following 141 assumptions on S ( R,  ) and Q ( R,  ) .
Assumption 1.
(i) S ( R,  ) is twice-differentiable in ( R,  ) .
(ii) r  S
( R,  ) , r 2 R  S ( R,  ) ,r 2  S ( R,  ) , r R Q ( R,  ) , and r  Q ( R,  ) are Lipschitz continuous w.r.t.
with constants L s , L s, 1 ,
L s, 2 , L q, 1 , and L q, 2 .
(iii) r 2 R  S ( R,  ) , r 2  S ( R,  ) , and r  Q ( R,  ) are Lipschitz continuousw.r.t.
R with constants L s, 3 , and L s, 4 , and L q, 3 .
(iv) S ( R,  ) are µ s -strongly convex in  .
(v)
[Equation]
and kr
[Equation]
Given  ⇤ ( R ) , the gradient of upper-level objective u ( R ) ⌘ Q ( R,  ⇤ ( R )) can be obtained as
[Equation]
In [21], it is shown that the outer gradient is the stochastic estimate of
[Equation]
However, this involves computing the Hessian and is expensive.
The proposed algorithm uses 150 h kq ⌘ r R Q ( R k ,  k +1 ; ⇡ k ) , which is the stochastic estimate of r R Q ( R k ,  k +1 ) .
Note thatr R Q ( R k ,  k +1 ) is the ﬁrst-order approximation of the outer gradient in (10).
This approxima- 152 tion greatly reduces the time and memory complexities, but leads to a bias that can be bounded by aconstant.
[Equation]
Next, we also make the following assumption similar to [21].
Let h ks ⌘ r  S ( R k ,  k ; ⇠ k ) .
Assumption 2.
For any k  0 , there exist constants  s ,  q , and b q such that:
[Equation]
Denote the expected gap between  k and the optimal network parameter given reference vectors 157 R k  1 by  k  ⌘ E ⇠ [ k  k   ⇤
( R k  1 ) k ] .
Similarly, denote the gap between R k and the optimalreference vectors R ⇤ in (8) by  kR ⌘ E ⇡ [ k R k  R ⇤ k ] .
Theorem 1.
With Assumptions 1–2, assume that u ( R ) is µ q -strongly convex, and the step sizes 160 ( ↵ k ,  k ) satisfy
[Equation]
where L, L q are constants and c 0 , c 1 > 0 are free parameters.
For any k  1 , the iterates generated 162 by Algorithm 1 satisfy
[Equation]
where . denotes that numerical constants are omitted.
With diminishing step sizes ↵ k = c ↵ / ( k + k ↵ ) ,  k =
c  / ( k + k  ) 2 / 3 , wherek ↵ = max
[Equation]
the following Corollary shows that the assumptions (12a) and (12b) in Theorem 1 are satiﬁed, and 166 thus  k converges to the optimal solution  ⇤ ( R k  1 ) .
Corollary 1.
k converges to the optimal  ⇤ ( R k  1 ) of (9).
Since the minimizer of linear scalarization function is Pareto-optimal for the original multi-objective 169 optimization problem [5],  k converges to a Pareto-optimal solution.
Experiments
[Figure] In this section, extensive experiments are performed, including synthetic problems (Section 4.1), 172 multi-task learning (Section 4.2), accuracy-fairness trade-off (Section 4.3), and usage on largernetworks (Section 4.4).
Finally, ablation study is presented in Section 4.5.
4.1 Synthetic Problems
Experiments are performed on four commonly used multi-objective benchmark problems [10, 49, 31] 176 with different numbers of objectives: (i) 2-objective DTLZ2 [11], (ii) 3-objective DTLZ2 [11], (iii)2-objective scaled-DTLZ2 [11], and (iv) 3-objective MaF1 [7].
Their detailed deﬁnitions are in 178 Appendix B.1.
The PFs of problems (i) and (ii) are regular, while those of (iii) and (iv) are irregular.
The number of inputs is set to 30.
We aim to get 15 non-dominated solutions for each 2-objective 180 problem, and 36 non-dominated solutions for each 3-objective problem.
The proposed algorithm 1 (GMOOAR-U using quality function (6) and GMOOAR-HV using qualityfunction (7))
is compared with the state-of-the-art COSMOS [41],which uses ﬁxed reference vectors.
183 For the 2-objective problems, reference vectors for COSMOS are generated by following theirstrategy in [41].
As 3-objective problems are not considered in [41], we generate reference vectors 185 for COSMOS by the method in [9].
For GMOOAR, the reference vectors are initialized randomly.
As in [37], a neural network (with 2 hidden layers, each with 20 units) is used.
More experimental 187 details can be found in Appendix B.2.
Figure 3 shows the solution sets obtained by GMOOAR-U on 3-objective DTLZ2 when the region of 189 interest is a subspace R constrained as in (5).
Note that this constraint can be difﬁcult for COSMOS 190 as it is unclear how the pre-speciﬁed RVs can be uniformly generated in R .
Moreover, even with a 191 set of uniformly distributed RVs, they may not lead to a uniformly distributed set of solutions.
4.2 Multi-Task Learning
In this experiment, we use three benchmark datasets from [28]: Multi-MNIST, Multi-Fashion, 194 and Multi-Fashion+MNIST.
In Multi-MNIST, each image is constructed by putting two different
MNIST images together, one at the bottom-right (BR) and the other at the top-left (TL).
Similarly, 196 Multi-Fashion images are constructed by combining images from FashionMNIST [47], while MultiFashion+MNIST images are constructed by combining one MNIST image with one FashionMNIST 198 image.
More details can be found in [28].
The goal is to classify both the BR and TL images correctly,by minimizing the two cross-entropy losses using a single neural network.
As in [28, 34, 41], we use 200 the LeNet [25] with multi-head as base network.
We compare the proposed algorithms (GMOOAR-U and GMOOAR-HV) with (i) EPO [34], (ii) {+
+}
Pareto hypernetworks (PHN-LS and PHN-EPO) [37], (iii) MOO-SVGD [31], and (iv) COSMOS[41].
For EPO, PHN-LS, PHN-EPO and COSMOS, we generate reference vectors following the 204 strategy in [41].
For GMOOAR, the reference vectors are initialized randomly.
The experiment isrepeated 10 times with different random seeds.
Following common practice [41], we obtain a set of n solutions 2 in each iteration.
They are evaluatedon the validation set every 5 epochs.
We only keep the solutions of iteration k best as the ﬁnal solution 208 set, where k best is the iteration that yields the solution set with the largest validation HV.
Note that 209 the original implementation of MOO-SVGD (obtained from the authors) stores all non-dominatedsolutions of each iteration in an archive A .
On termination, they try all size- n subsets of A and selectthe subset with the largest HV on the validation set as the ﬁnal solution set.
As there are C |A| n such 212 subsets and |A| is large when MOO-SVGD terminates, this can be very expensive.
In order to be fair 213 to all algorithms being compared, we thus also use the aforementioned commonly practiced strategyon MOO-SVGD.
Figure 4 shows the test losses obtained by the solution set with median HV over the 10 runs.
As can 216 be seen, on Multi-MNIST and Multi-Fashion, the solution sets obtained by COSMOS are dense inthe middle but sparse towards the ends, while the solution sets obtained by GMOOAR-U are more
[Figure]
[Figure]
uniform.
Moreover, compared to all other baselines, the solution sets obtained by GMOOAR are 219 closer to the bottom-right corner where the underlying true PF resides.
For MOO-SVGD, many ofits obtained solutions are much inferior, and only one of them is in the range shown in Figure 4.
A 221 complete plot of all the MOO-SVGD solution sets is in Appendix C.
Table 1 shows the average HV’s of the solution sets over the 10 runs.
As can be seen, GMOOAR 223 consistently outperforms all others.
Moreover, the number of parameters of GMOOAR is comparable
[Figure]
[Table]
to that of
, and is much fewer than the other baselines.
Compared to the network, has only 37% more parameters.
4.3 Accuracy-Fairness Tradeoff
In this experiment, we follow [41] and aim to achieve both high accuracy and fairness on three tabular 228 datasets: Adult [15], Compass [1] [Table] and Default [48].
The accuracy is measured by the cross-entropyloss, while fairness is measured by a hyperbolic tangent relaxation of the Difference of Equality 230 of Opportunity (DEO) [38].
As in [37, 41], a 2-hidden-layer multilayer perceptron is used as basenetwork.
More details can be found in Appendix B.4.
The experiment is repeated 10 times.
Figure 5 shows the test loss and fairness measure obtained by the solution set with median HV over 233 the 10 runs.
Since the datasets are not difﬁcult, the approximated PFs obtained by various algorithmsare close.
Solutions obtained by GMOOAR are uniformly distributed, while those obtained by 235 COSMOS, EPO, PHN-LS, and PHN-EPO are very dense in the top-left region.
Table 2 shows the HVvalues of the obtained datasets.
It can be seen that GMOOAR achieves better HVs than the baselines.
4.4 Larger Networks
To demonstrate that the proposed method can be used on larger networks, we apply GMOOAR on 239 the EfﬁcientNet-B4 [43] with about 17 million parameters.
Following [41], we perform experiments
[Figure]
[Table]
on two easy tasks ("Goatee" and "Mustache") and two hard tasks ("Oval Face" and "Pointy Nose") 241 from the 32 tasks in CelebA [32].
Since EfﬁcientNet-B4 is around 400 times larger than the LeNetused in previous experiments, PHN-LS, PHN-EPO, and MOO-SVGD cannot be run on our machine 243 (with RTX-2080Ti and 11GB memory).
For performance evaluation, the testing cross-entropy loss ofeach selected task is used.
The experiment is repeated 5 times with different random seeds.
Figure 6 shows the test losses obtained by the solution set with median HV over the 5 runs.
The 246 corresponding HV values are shown in Table 7.
On CelebA-Easy, both GMOOAR-U and GMOOARHV outperform COSMOS in terms of HV.
On CelebA-Hard, all three algorithms achieve similar 248 HVs, though that of GMOOAR-HV is slightly better. [Equation]
[Figure] [Figure]
4.5 Ablation Study
In this experiment, we study the effect of RV learning rate ↵ and bandwidth h in (6) on the perfor- 251 mance of GMOOAR-U.
We use the same setting as in Section 4.3.
The experiment is repeated 10times with different random seeds.
[Figure]
Figure 8a shows the variations of HV with ↵ ( h is ﬁxed to 0 . 01 ).
As can be seen, ↵ too small results in almost no RV adaption and thus poor performance, while ↵ too large may lead to unstable learning.
Figure 8b shows the variation of HV with h ( ↵ is ﬁxed to 0 .
When h is too small (resp. too large), Q ( R,  ) is close to zero (resp. n ) and the gradient vanishes, making learning difﬁcult.
Conclusion
In this paper, we present a novel gradient-based MOO algorithm with adaptive RVs.
The proposed 255 algorithm can efﬁciently adapt the RVs during optimization and provide diverse solutions with a smalloverhead compared to single-objective optimization.
Experiments show the ability of the proposed 257 strategies to obtain well-distributed solutions based on the speciﬁed quality function.
Incorporatingother state-of-the-art MOO algorithms into our model can be an interesting direction.