Multi-Objective Deep Learning with Adaptive Reference Vectors
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Many deep learning models involve optimizing multiple objectives.
Since objectives are often conﬂicting, we aim to get diverse and representative trade-offsolutions among these objectives.
Gradient-based multi-objective optimization(MOO) algorithms using reference vectors have shown promising performance.
However, they may still produce undesirable solutions due to mismatch betweenthe pre-speciﬁed reference vectors and the problem’s underlying Pareto front.
Inthis paper, we propose a novel gradient-based MOO algorithm with adaptive reference vectors.
We formulate reference vector adaption as a bilevel optimizationproblem, and solve it with an efﬁcient solver.
Theoretical convergence analysis isalso provided.
Experiments on an extensive set of learning scenarios demonstratethe superiority of the proposed algorithm over the state-of-the-art.
Introduction
Deep learning models are often evaluated under multiple, potentially conﬂicting, criteria.
For example,in multi-task learning [6], a single model is required to perform well on multiple tasks.
In somescenarios, besides accuracy, model fairness is also important so as to ensure that the model is notbiased against gender and race.
These problems can all be formulated as multi-objective optimization(MOO)
[36] problems, and have attracted attention from various ﬁelds such as energy resourceoptimization [8] and signal processing [4].
Since the multiple objectives usually cannot be optimized simultaneously by a single solution, thegoal of MOO is to ﬁnd a set of solutions with different trade-offs to approximate the true Pareto front(PF).
The past decades have witnessed the birth of a large number of gradient-free MOO algorithms,such as evolutionary multi-objective optimization algorithms [10, 49] and Bayesian multi-objectivealgorithms [23, 3].
These algorithms perform well on small-scale problems but fail to provide usefulsolutions when facing the huge number of parameters in deep learning models.
Recently, gradient-based MOO algorithms [13] have demonstrated promising performance in deeplearning.
Based on the work in [42]
Mahapatra et
[34] propose togenerate multiple solutions using a set of reference vectors (RV).
These algorithms try to search forsolutions on the PF closest to each RV.
Some strategies are further proposed to improve the efﬁciencyby training a hypernetwork [37, 27] or reference-vector-conditioned network [41].
However, notethat a set of uniformly distributed RVs may not lead to a set of uniformly distributed solutions.
Hence,an important limitation of these algorithms is that solutions generated using ﬁxed RVs may not coversome parts of the PF, thus failing to provide enough information about the PF.
As an illustration, in Figure 1a, solutions obtained by the ﬁxed RVs are close to the ends of the PF. If the RVs are properlypositioned, it is possible that the obtained solution set can uniformly cover the PF (Figure 1b).
[Figure]
Since it is impossible to know the true PF of the problem before optimization, how to generate a proper RV set is the main challenge.
In this paper, we propose to learn the set of RVs simultaneously with the model parameters.
This is formulated as a bilevel optimization problem [12], in which the lower-level optimization problem obtains the Pareto-optimal solutions with the given set of RVs, while the upper-level problem optimizes the RVs based on some quality measure.
Moreover, we propose an inexpensive solver for this optimization problem, while still showing nice theoretical convergence properties.
Experiments on an extensive set of learning scenarios demonstrate the superiority of the proposed
Related Work: Multi-Objective Optimization (MOO)
In multi-objective optimization (MOO) [36], one aims to minimize m ≥ 2 objectives{ f 1 ( φ ) , f 2 ( φ ) , . . .
f m ( φ ) } , or, equivalently, the vector-valued function:
[Equation]
A solution φ 1 is dominated by another solution φ 2 if and only if f i ( φ 1 ) ≥ f i ( φ 2 ) for i ∈ [ m ] ≡{ 1 , . . .
, m } , and ∃ i ∈ [ m ] , f i ( φ 1 ) >
f i ( φ 2 ) .
A solution φ ∗ is Pareto-optimal if and only if it is notdominated by any other φ (cid:48) .
A Pareto front (PF) is the set of multi-objective values of all Paretooptimal solutions.
A PF is regular [45] if its shape is simplex-like (i.e., all vectors with positivedirections intersect it when they start at the origin), and irregular otherwise.
It is shown that regular
PFs are not very realistic [22].
Since the number of Pareto-optimal solutions is usually large or even inﬁnite, a set of n Paretooptimal solutions Φ = { φ 1 , . .
φ n } is often used to approximate the PF.
Denote the correspondingmulti-objective values as F = { f ( φ 1 ) , . . .
, f ( φ n ) } .
The quality of Φ can be evaluated from twoperspectives: convergence and diversity [24].
Convergence refers to the distance between F and thetrue PF, while diversity measures whether the solutions are well-distributed in the space of objectives.
A popular measure, which evaluates both convergence and diversity, is the hypervolume (HV) [50].
Given a reference point z ∈ R m , the HV of F is:
[Equation]
where [ z, f i ] ≡ { q ∈ R m | f i ≤ q ≤ z } , and λ ( · ) is the Lebesgue measure of a set.
A reference vector (RV) [49], sometimes called the weight vector or preference vector, can be used toguide the optimization algorithm by indicating the preferred point on the PF.
Usually, an algorithm isexpected to obtain the Pareto-optimal solution closest to the given RV in the objective space.
Givena RV r ∈ R m + , the MOO problem in (1) can be converted to a single-objective problem by using ascalarization function s ( φ ; r ) .
The most straightforward construction is linear scalarization:s ( φ ; r ) = m (cid:88) j
[Equation]
r ( [Equation] f j ( φ ) , (3) which weights the m objectives with the elements r ( j ) ’s of r .
It is known that the minimizer ofs ( φ ; r ) is also Pareto-optimal for the original problem in (1) (Ch 4.7, [5]).
However, the solutionobtained may be far way from the given RV.
To encourage the solutions to be closer to this vector, apenalty term can be added to the linear scalarization function [41]:
[Equation]
where γ is a constant.
Given a (discrete) set of solutions, Ma et al.
[33] generates explorationdirections to spawn new solutions on the PF, leading to a continuous PF.
By using n RVs from a subset R ⊆ R m × n + , a set of solutions can be obtained to approximate theentire PF.
Usually, R is simply set to R m × n + .
However, sometimes the decision-makers may only beinterested in a speciﬁc region of the PF [2].
For example, we may limit the angle between any RVand each coordinate axis.
R can then be changed to:
[Equation]
where u i is the i th coordinate axis and ϕ 1 , ϕ 2 are the maximum and minimum allowable angles,respectively.
However, note that the set of pre-speciﬁed RVs may not ﬁt the problem’s PF (e.g., someof them may not intersect the underlying PF), leading to an undesirable solution distribution.
Gradient-Free MOO.
Evolutionary MOO algorithms (e.g., NSGA-II[10], MOEA/D[49]) and Bayesian MOO algorithms (e.g., BMOA[23], USeMO[3]) are widely used for small-scale black-boxproblems.
These algorithms assume that gradient information is not available.
Hence, they often failto converge on deep learning problems where the solution space can be very large.
Gradient-Based MOO.
Gradient-based MOO algorithms are more efﬁcient when problems havedifferentiable objectives and a large number of parameters.
Sener and Koltun [42] propose to apply
Multiple-Gradient Descent Algorithm (MGDA) [13] to multi-task learning.
Liu and Vicente [30]provide theoretical analysis of stochastic MGDA.
MGDA can be further extended to incorporate
RVs (e.g., EPO [34, 35], ParetoMTL [28]).
Some algorithms (e.g., MOO-SVGD [31], HIGA [46])can output a solution set without using RVs.
However, they optimize several neural networkssimultaneously, and so are computationally expensive and need large GPU memory (especially whenthe neural network is large).
Proposed Algorithm
As mentioned in Section 1, the ﬁxed uniformly distributed RVs used in common practice may resultin undesirable solution distributions.
Instead of using a ﬁxed set of n RVs ( R =
[ r 1 , . .
, r n ] ∈ R ),we propose to adapt them so that the resultant solution set is well-distributed.
In Section 3.1, weﬁrst introduce a reference vector-conditioned neural network so that reference vectors can be easilyhandled without using a lot more parameters.
Section 3.2 then formulates reference vector adaptationas a bilevel optimization problem, and an efﬁcient solver is proposed in Section 3.3.
Its convergenceproperties are then studied in Section 3.
3.1 Reference Vector-Conditioned Neural Network
In deep learning models, φ corresponds to the network parameters and optimizing Φ = { φ 1 , . .
. , φ n }means optimizing n neural networks (as in EPO [34] and MOO-SVGD [31]), which is highlyinefﬁcient.
To alleviate this problem in deep MOO algorithms, Navon et al.
[37] and Lin et al.
[27]propose to train a single hypernetwork [20] that can output neural network parameters based onthe RV.
In particular, the Pareto Hypernetwork (PHN) in [37] proposes two ways to optimize thehypernetwork, leading to (i) PHN-LS, which uses linear scalarization, and (ii) PHN-EPO, whichuses EPO.
However, the hypernetwork incurs signiﬁcant computational overhead.
For example,the hypernetwork in [37] is around 100 times larger than the base neural network.
In this paper,we use the more efﬁcient conditioned network [14, 41].
Speciﬁcally, Ruchte and Grabocka [41]concatenates the RV and data sample, and treat this as a joint input to the network.
On the otherhand, YOTO [14] is originally developed for use with a family of parameterized loss functions, inwhich the loss parameter is incorporated with the sample into the network via FilM layers [39].
Itis theoretically shown that YOTO is as powerful as using n neural networks [14].
In this work, weadapt the architecture in [14] by replacing the loss parameter with the RV.
In this way, the proposedmodel only has a small parameter overhead compared to the single deep network.
3.2 Reference Vector Adaption via Bilevel Optimization
Consider a RV-conditioned neural network f ( φ, r ) with parameter φ and RV r as input.
Witha set of RVs R =
[ r 1 , . . . , r n ] , the multi-objective values of a solution set can be written as[ f ( φ, r 1 ) , . . .
, f ( φ, r n )] .
We use a function ˆ Q ( · ) to measure its quality.
Two choices are considered in this paper.
The ﬁrst one encourages the f ( φ, r i ) ’s are far away from each other (and thus more uniformly distributed in the space of objectives):
[Equation]
where h is a constant.
The second one is:
[Equation]
which encourages the maximization of HV in (7).
Note that the HV-optimal solution is usually notuniformly distributed [19, 44].
To obtain the set of RVs R that generates φ , we formulate it as a bilevel optimization problem [12].
Recently, bilevel optimization has gained great popularity in many machine learning problems suchas meta-learning [16], neural architecture search [29], and hyperparameter optimization [17].
Weconsider the following bilevel optimization problem:
[Equation]
where Q ( R, φ ∗ ( R )) ≡ − ˆ Q ( R, φ ∗ ( R )) , S ( R, φ ) ≡ (cid:80) ni
=1 s ( φ ; r i ) , and s ( φ ; r ) is the scalarizationfunction in (3) or (4).
The lower-level optimization problem (9) obtains the Pareto-optimal solutionswith the given set of RVs R , while the upper-level optimization problem (8) optimizes R to maximizethe corresponding solution quality.
The idea of RV adaption is also used in some evolutionary algorithms [40, 26].
However, they updatethe RVs using information from the current population and archive, and cannot be directly used ingradient-based MOO algorithms.
Moreover, they do not have any theoretical guarantees.
3.3 Solving the Bilevel Optimization Problem
There are various bilevel optimization solvers for (8).
Many of them involve propagation through theinner loop [16, 18], which has a large computational overhead compared to [41].
Here, we proposean inexpensive solver that still has theoretical guarantees (Section 3.4).
Essentially, it performs onlyone stochastic gradient descent step in both the inner and outer loops.
The proposed procedure, shown in Algorithm 1, is called Gradient-based Multi-Objective Optimization algorithm with Adaptive Reference vectors (GMOOAR).
In each iteration k , minibatches ξ k and π k are randomly sampled from the data and then used to estimate the stochasticgradients ∇ φ S ( R k , φ k ; ξ k ) and ∇ R Q ( R k , φ k +1 ; π k ) .
proj R ( · ) is the Euclidean projection operatoronto R that ensures that RVs are inside R .
In the sequel, the algorithm using uniformity-relatedquality function (6) is denoted GMOOAR-U, while the one using (7) is denoted GMOOAR-HV.
The proposed algorithm has O ( w + m ) memory and O ( w + m ) time complexity per iteration, wherew is the dimension of φ .
Since m (cid:28) w in most cases, the proposed algorithm has comparable timeand space complexity with COSMOS, which is O ( w ) . 3.4 Convergence
In this section, we provide convergence analysis for Algorithm 1.
As in [21], we make the followingassumptions on S ( R
Assumption 1.
(i) S ( R, φ ) is twice-differentiable in ( R, φ ) .
[Equation]
∇ 2 φφ S
R, φ ) , ∇ R Q ( R, φ ) , and ∇ φ Q ( R, φ ) are Lipschitz continuous w.r.t.
φ with constants L s , L s, 1 ,
L s, 2 , L q, 1 , and L q, 2 .
(iii) ∇ 2 Rφ S ( R, φ ) , ∇ 2 φφ S R, φ ) , and ∇ φ Q ( R, φ ) are Lipschitz continuousw.r.t.
R with constants L s, 3 , and L s, 4 , and L q, 3 .
(iv) S ( R, φ ) are µ s -strongly convex in φ .
(v)
[Equation]
and (cid:107)∇
[Equation]
, φ ) (cid:107) ≤ C q .
Given φ ∗ ( R ) , the gradient of upper-level objective u ( R ) ≡ Q ( R, φ ∗ ( R )) can be obtained as
[Equation]
In [21], it is shown that the outer gradient is the stochastic estimate of
[Equation]
However, this involves computing the Hessian and is expensive.
The proposed algorithm uses [Equation] kq ≡ ∇ R Q ( R k , φ k +1 ; π k ) , which is the stochastic estimate of ∇ R Q ( R k , Φ k +1 ) .
Note that∇ R Q ( R k , Φ k +1 ) is the ﬁrst-order approximation of the outer gradient in (10).
This approximation greatly reduces the time and memory complexities, but leads to a bias that can be bounded by aconstant.
[Equation]
Next, we also make the following assumption similar to [21].
Let h ks ≡ ∇ φ S ( R k , φ k ; ξ k ) .
Assumption 2.
For any k ≥ 0 , there exist constants σ s , σ q , and b q such that:
[Equation]
Denote the expected gap between φ k and the optimal network parameter given reference vectors
R k − 1 by ∆ kφ ≡ E ξ [ (cid:107) φ k − φ ∗ ( R k − 1 ) (cid:107) ] .
Similarly, denote the gap between R k and the optimalreference vectors R ∗ in (8) by ∆ kR ≡ E π [ (cid:107) R k − R ∗ (cid:107) ] .
Theorem 1.
With Assumptions 1–2, assume that u ( R ) is µ q -strongly convex, and the step sizes( α k , β k ) satisfy
[Equation]
where L, L q are constants and c 0 , c 1 > 0 are free parameters.
For any k ≥ 1 , the iterates generatedby Algorithm 1 satisfy
[Equation]
where (cid:46) denotes that numerical constants are omitted.
With diminishing step sizes α k = c α / ( k + k α ) , β k =
c β / ( k + k β ) 2 / 3 , where
[Equation]
the following Corollary shows that the assumptions (12a) and (12b) in Theorem 1 are satiﬁed, andthus φ k converges to the optimal solution φ ∗ ( R k − 1 ) .
Corollary 1.
φ k converges to the optimal φ ∗ ( R k − 1 ) of (9).
Since the minimizer of linear scalarization function is Pareto-optimal for the original multi-objectiveoptimization problem [5], φ k converges to a Pareto-optimal solution.
Experiments
In this section, extensive experiments are performed, including synthetic problems (Section 4.1),multi-task learning (Section 4.2), accuracy-fairness trade-off (Section 4.3), and usage on largernetworks (Section 4.4).
Finally, ablation study is presented in Section 4.5.
4.1 Synthetic Problems
Experiments are performed on four commonly used multi-objective benchmark problems [10, 49, 31]with different numbers of objectives: (i) 2-objective DTLZ2 [11], (ii) 3-objective DTLZ2 [11], (iii)2-objective scaled-DTLZ2 [11], and (iv) 3-objective MaF1 [7].
Their detailed deﬁnitions are in Appendix B.1.
The PFs of problems (i) and (ii) are regular, while those of (iii) and (iv) are irregular.
The number of inputs is set to 30.
We aim to get 15 non-dominated solutions for each 2-objectiveproblem, and 36 non-dominated solutions for each 3-objective problem.
The proposed algorithm 1 (GMOOAR-U using quality function (6) and GMOOAR-HV using qualityfunction (7))
is compared with the state-of-the-art COSMOS [41],which uses ﬁxed reference vectors.
For the 2-objective problems, reference vectors for COSMOS are generated by following theirstrategy in [41].
As 3-objective problems are not considered in [41], we generate reference vectorsfor COSMOS by the method in [9].
For GMOOAR, the reference vectors are initialized randomly.
As in [37], a neural network (with 2 hidden layers, each with 20 units) is used.
More experimentaldetails can be found in Appendix B.2.
Figure 3 shows the solution sets obtained by GMOOAR-U on 3-objective DTLZ2 when the region ofinterest is a subspace R constrained as in (5).
Note that this constraint can be difﬁcult for COSMOSas it is unclear how the pre-speciﬁed RVs can be uniformly generated in R .
Moreover, even with aset of uniformly distributed RVs, they may not lead to a uniformly distributed set of solutions.
4.2 Multi-Task Learning
In this experiment, we use three benchmark datasets from [28]: Multi-MNIST, Multi-Fashion,and Multi-Fashion+MNIST.
In Multi-MNIST, each image is constructed by putting two different
MNIST images together, one at the bottom-right (BR) and the other at the top-left (TL).
Similarly, Multi-Fashion images are constructed by combining images from FashionMNIST [47], while MultiFashion+MNIST images are constructed by combining one MNIST image with one FashionMNISTimage.
More details can be found in [28].
The goal is to classify both the BR and TL images correctly,by minimizing the two cross-entropy losses using a single neural network.
As in [28, 34, 41], we usethe LeNet [25] with multi-head as base network.
We compare the proposed algorithms (GMOOAR-U and GMOOAR-HV) with (i) EPO [34], (ii)[-
-]
Pareto hypernetworks (PHN-LS and PHN-EPO) [37], (iii) MOO-SVGD [31], and (iv) COSMOS[41].
For EPO, PHN-LS, PHN-EPO and COSMOS, we generate reference vectors following thestrategy in [41].
For GMOOAR, the reference vectors are initialized randomly.
The experiment isrepeated 10 times with different random seeds.
Following common practice [41], we obtain a set of n solutions 2 in each iteration.
They are evaluatedon the validation set every 5 epochs.
We only keep the solutions of iteration k best as the ﬁnal solutionset, where k best is the iteration that yields the solution set with the largest validation HV.
Note thatthe original implementation of MOO-SVGD (obtained from the authors) stores all non-dominatedsolutions of each iteration in an archive A .
On termination, they try all size- n subsets of A and selectthe subset with the largest HV on the validation set as the ﬁnal solution set.
As there are C |A| n suchsubsets and |A| is large when MOO-SVGD terminates, this can be very expensive.
In order to be fairto all algorithms being compared, we thus also use the aforementioned commonly practiced strategyon MOO-SVGD.
Figure 4 shows the test losses obtained by the solution set with median HV over the 10 runs.
As canbe seen, on Multi-MNIST and Multi-Fashion, the solution sets obtained by COSMOS are dense inthe middle but sparse towards the ends, while the solution sets obtained by GMOOAR-U are more We [Figure] HV=
[Figure]
COSMOS
[Figure]
HV
[Figure]
[Figure]
uniform.
Moreover, compared to all other baselines, the solution sets obtained by GMOOAR arecloser to the bottom-right corner where the underlying true PF resides.
For MOO-SVGD, many ofits obtained solutions are much inferior, and only one of them is in the range shown in Figure 4.
Acomplete plot of all the MOO-SVGD solution sets is in Appendix C.
Table 1 shows the average HV’s of the solution sets over the 10 runs.
As can be seen, GMOOARconsistently outperforms all others.
Moreover, the number of parameters of GMOOAR is comparable
[Figure]
[Figure]
[Figure]
Runtime(s)
[Figure]
MNIST
Multi- [Figure] Fashion-MNIST EPO [34] 2.95 ± 0.
2.31 ± 0.
2.86 ± 0.
PHN-EPO [37] 2.82 ± 0.
2.16 ± 0.
2.74 ± 0.
PHN-LS [37] 2.79 ± 0.
2.14 ± 0.
2.67 ± 0. MOO-SVGD [31] 2.67 ± 0. 2.02 ± 0. 2.54 ± 0.
COSMOS [41] 2.95 ± 0.
2.31 ± 0.
2.82 ± 0. GMOOAR [Table] to that of COSMOS, and is much fewer than the other baselines.
Compared to the base network, GMOOAR has only 37% more parameters.
4.3 Accuracy-Fairness Trade-off
In this experiment, we follow [41] and aim to achieve both high accuracy and fairness on three tabulardatasets: Adult [15], Compass [1], and Default [48].
The accuracy is measured by the cross-entropyloss, while fairness is measured by a hyperbolic tangent relaxation of the Difference of Equalityof Opportunity (DEO) [38].
As in [37, 41], a 2-hidden-layer multilayer perceptron is used as basenetwork.
More details can be found in Appendix B.4.
The experiment is repeated 10 times.
Figure 5 shows the test loss and fairness measure obtained by the solution set with median HV overthe 10 runs.
Since the datasets are not difﬁcult, the approximated PFs obtained by various algorithmsare close.
Solutions obtained by GMOOAR are uniformly distributed, while those obtained by COSMOS, EPO, PHN-LS, and PHN-EPO are very dense in the top-left region.
Table 2 shows the HVvalues of the obtained datasets.
It can be seen that GMOOAR achieves better HVs than the baselines.
4.4 Larger Networks
To demonstrate that the proposed method can be used on larger networks, we apply GMOOAR onthe EfﬁcientNet-B4 [43] with about 17 million parameters.
Following [41], we perform experiments
[Figure]
[Table]
on two easy tasks ("Goatee" and "Mustache") and two hard tasks ("Oval Face" and "Pointy Nose")from the 32 tasks in CelebA [32].
Since EfﬁcientNet-B4 is around 400 times larger than the LeNetused in previous experiments, PHN-LS, PHN-EPO, and MOO-SVGD cannot be run on our machine(with RTX-2080Ti and 11GB memory).
For performance evaluation, the testing cross-entropy loss ofeach selected task is used.
The experiment is repeated 5 times with different random seeds.
Figure 6 shows the test losses obtained by the solution set with median HV over the 5 runs.
Thecorresponding HV values are shown in Table 7.
On CelebA-Easy, both GMOOAR-U and GMOOARHV outperform COSMOS in terms of HV.
On CelebA-Hard, all three algorithms achieve similar HVs, though that of GMOOAR-HV is slightly better.
[Figure] [Figure]
4.5 Ablation Study
[Figure]
In this experiment [Figure] we study the effect of RV learning rate α and bandwidth h in (6) on the performance of GMOOAR-U.
We use the same setting as in Section 4.3.
The experiment is repeated 10times with different random seeds
[Figure]
-U using the Compass dataset.
Figure 8a shows the variations of HV with α ( h is ﬁxed to 0 . 01 ).
As can be seen, α too small results in almost no RV adaption and thus poor performance, while α too large may lead to unstable learning.
Figure 8b shows the variation of HV with h ( α is ﬁxed to 0 . 005 ).
When h is too small (resp. too large), Q ( R, φ ) is close to zero (resp. n ) and the gradient vanishes, making learning difﬁcult.
Conclusion
In this paper, we present a novel gradient-based MOO algorithm with adaptive RVs.
The proposedalgorithm can efﬁciently adapt the RVs during optimization and provide diverse solutions with a smalloverhead compared to single-objective optimization.
Experiments show the ability of the proposedstrategies to obtain well-distributed solutions based on the speciﬁed quality function.
Incorporatingother state-of-the-art MOO algorithms into our model can be an interesting direction.