C ONTROLLABLE P ARETO M ULTI -T ASK L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
A multi-task learning (MTL) system aims at solving multiple related tasks at the same time.
With a ﬁxed model capacity, the tasks would be conﬂicted with each other, and the system usually has to make a trade-off among learning all of them together.
Multiple models with different preferences over tasks have to be trained and stored for many real-world applications where the trade-off has to be made online.
This work proposes a novel controllable Pareto multi-task learning framework, to enable the system to make real-time trade-off switch among different tasks with a single model.
To be speciﬁc, we formulate the MTL as a preference-conditioned multiobjective optimization problem, for which there is a parametric mapping from the preferences to the Pareto stationary solutions.
A single hypernetwork-based multi-task neural network is built to learn all tasks with different trade-off preferences among them, where the hypernetwork generates the model parameters conditioned on the preference.
At the inference time, MTL practitioners can easily control the model performance based on different trade-off preferences in real-time.
Experiments on different applications demonstrate that the proposed model is efﬁcient for solving various multi-task learning problems.
I NTRODUCTION
MTL allows practitioners to control the trade-offs among tasks in real time with a single model, which could be desirable for many
MTL applications
Multi-task learning (MTL) is important for many real-world applications, such as computer vision (Kokkinos, 2017), natural language processing (Subramanian et al., 2018), and reinforcement learning (Van Moffaert & Now ´ e, 2014).
In these problems, multiple tasks are needed to be learned at the same time.
An MTL system usually builds a single model to learn several related tasks together, in which the positive knowledge transfer could improve the performance for each task.
In addition, using one model to conduct multiple tasks is also good for saving storage costs and reducing the inference time, which could be crucial for many applications (Standley et al., 2020).
However, with ﬁxed learning capacity, different tasks could be conﬂicted with each other, and can not be optimized simultaneously (Zamir et al., 2018).
The practitioners might need to carefully design and train the tasks into different groups to achieve the best performance (Standley et al., 2020).
A considerable effort is also needed to ﬁnd a set of suitable weights to balance the performance of each task (Kendall et al., 2018;
Chen et al., 2018b; Sener & Koltun, 2018).
For some applications, it might need to train and store multiple models for different trade-off preferences among the tasks (Lin et al., 2019).
In many real-world MTL applications, the system will need to make a trade-off among different tasks in real time, and it is desirable to have the whole set of optimal trade-off solutions.
For example, in a self-driving system, multiple tasks must be conducted simultaneously but also compete for a ﬁxed resource (e.g., ﬁxed total inference time threshold), and their preferences could change in real time for different scenarios (Karpathy, 2019).
A recommendation system needs to balance multiple criteria among different stakeholders simultaneously, and making trade-off adjustment would be a crucial component (Milojkovic et al., 2019).
Consider the huge storage cost, it is far from ideal to train and store multiple models to cover different trade-off preferences, which is also not good for real-time adjustment.
In this paper, we propose a novel controllable Pareto multi-task learning framework, to learn the whole trade-off curve for all tasks with a single model.
As shown in Fig.
1, MTL practitioners can easily control the trade-off among tasks based on their preferences.
To our best knowledge, this is the ﬁrst approach to learn the MTL trade-off curve.
The main contributions are:
• We formulate solving an MTL problem as a preference-conditioned multiobjective optimization problem, and propose a novel Pareto solution generator to learn the whole tradeoff curve of Pareto stationary solutions.
The proposed Pareto solution generator is also a novel contribution to multiobjective optimization.
• We propose a general hypernetwork-based multi-task neural network framework, and develop an end-to-end optimization algorithm to train a single model concerning different trade-off preferences among all tasks simultaneously.
With the proposed model, MTL practitioners can control the trade-offs among different tasks in real time.
R ELATED W ORK
Multi-Task Learning.
The current works on deep multi-task learning mainly focus on designing novel network architecture and constructing efﬁcient shared representation among tasks (Zhang &
Yang, 2017; Ruder, 2017).
Different deep MTL networks, with hard or soft parameters sharing structures, haven been proposed in the past few years (Misra et al., 2016;
Long et al., 2017; Yang &
Hospedales, 2017).
However, how to properly combine and learn different tasks together remains a basic but challenging problem for MTL applications.
Although it has been proposed for more than two decades, the simple linear tasks scalarization approach is still the current default practice to combine and train different tasks in MTL problems (Caruana, 1997).
Some adaptive weight methods have been proposed to better combine all tasks in MTL problems with a single model (Kendall et al., 2018;
Chen et al., 2018b; Liu et al., 2019; Yu et al., 2020).
However, analysis on the relations among tasks in transfer learning (Zamir et al., 2018) and multitask learning (Standley et al., 2020) show that some tasks might conﬂict with each other and can not be optimized at the same time.
Sener and Koltun (Sener & Koltun, 2018) propose to treat MTL as a multiobjective optimization problem, and ﬁnd a single Pareto stationary solution among different tasks.
Pareto MTL (Lin et al., 2019) generalizes this idea, and proposes to generate a set of Pareto stationary solutions with different trade-off preferences.
Recent works focuses on generating diverse and dense Pareto stationary solutions (Mahapatra & Rajan, 2020; Ma et al., 2020).
Multiobjective Optimization.
Multiobjective optimization itself is a popular research topic in the optimization community.
Many gradient-based and gradient-free algorithms have been proposed in the past decades (Fliege & Svaiter, 2000; D´esid´eri, 2012; Miettinen, 2012).
In addition to MTL, they also can be used in reinforcement learning (Van Moffaert & Now´e, 2014) and neural architecture search (NAS) (Elsken et al., 2019).
However, most methods directly use or modify well-studied multiobjective algorithms to ﬁnd a single solution or a ﬁnite number of Pareto stationary solutions.
Recently, Lu et al.
(2020) proposed a novel supernet-based multi-objective neural architecture transfer framework.
This method simultaneously optimizes a set of neural networks, which are sampled from a large supernet, to approximate the optimal trade-off curve among different objectives via a single run.
The obtained supernet also supports fast adaption to new tasks and domains.
Some efforts have been made to learn the entire trade-off curve for different machine learning applications.
Parisi et al.
(2016) proposed to learn the Pareto manifold for a multi-objective reinforcement learning problem.
However, since this method does not consider the preference for generation, it does not support real-time preference-based adjustment.
Very recently, some efforts have been made to learn preference-based solution adjustment for multi-objective reinforcement learning (Yang et al., 2019) and image generation (Dosovitskiy & Djolonga, 2020) with simple linear combinations.
There is a concurrent work (Anonymous, 2021) that also proposes to learn the entire trade-off curve for
MTL problems by hypernetwork.
While this work emphasizes the runtime efﬁciency on training for multiple preferences, we highlight the beneﬁt of the real-time preference-based adjustment.
HyperNetworks.
The hypernetwork approach is initially proposed for dynamic modeling and model compression (Schmidhuber, 1992; Ha et al., 2017).
It also leads to various novel applications such as hyperparameter optimization (Brock et al., 2018; MacKay et al., 2019), Bayesian inference (Krueger et al., 2018; Dwaracherla et al., 2020), and transfer learning (von Oswald et al., 2020; Meyerson & Miikkulainen, 2019).
Recently, some discussions have been made on its initialization method (Chang et al., 2020), the optimization dynamic (Littwin et al., 2020), and its relation to other multiplicative interaction methods (Jayakumar et al., 2020).
This paper uses a hypernetwork to generate the weights of the main multi-task neural network conditioned on different preferences.
MTL AS M ULTI -O BJECTIVE O PTIMIZATION
An MTL problem involves learning multiple related tasks at the same time.
For training a deep multi-task neural network, it is also equal to minimize the losses for multiple tasks:
[Equation]
where θ is the neural network parameters and L i ( θ ) is the loss of the i -th task.
For learning all m tasks together, an MTL algorithm usually aims to minimize all the losses at the same time.
However, in many MTL problems, it is impossible to ﬁnd a single best solution to optimize all the losses simultaneously.
With different trade-offs among the tasks, the problem (1) could have a set of
Pareto solutions which satisfy the following deﬁnitions (Zitzler & Thiele, 1999):
Pareto dominance.
Let θ a , θ b be two solutions for problem (1), θ a is said to dominate θ b ( θ a ≺ θ b ) if and only if L i ( θ a ) ≤ L i ( θ b ) , ∀ i ∈ { 1 , ..., m } and L j ( θ a ) <
L j ( θ b ) , ∃ j ∈ { 1 , ..., m } .
Pareto optimality.
θ ∗ is a Pareto optimal solution if there does not exist ˆ θ such that ˆ θ ≺ θ ∗ .
The set of all Pareto optimal solutions is called the Pareto set.
The image of the Pareto set in the objective space is called the Pareto front.
Approximation with Finite Pareto Stationary Solutions.
The number of Pareto optimal solutions could be inﬁnite, and their objective values are on the boundary of the valid value region (Boyd & Vandenberghe, 2004).
Under mild conditions, the whole Pareto set and Pareto front would be ( m − 1) dimensional manifolds in the solution space and objective space, respectively (Miettinen, 2012).
In addition, for a general multiobjective optimization problem, no method can guarantee to ﬁnd a Pareto optimal solution.
Traditional multiobjective optimization algorithms aim at ﬁnding a set of ﬁnite Pareto stationary solutions to approximate the whole Pareto set:
[Equation]
where ˆ S is the approximated Pareto set of K estimated solutions, and ˆ F is the set of corresponding objective vectors in the objective space.
The current multiobjective optimization based MTL methods aim to ﬁnd a single ( K = 1 ) (Sener & Koltun,
P REFERENCE -C ONDITIONED P ARETO S OLUTION G ENERATOR
Instead of ﬁnite Pareto set estimation, we propose to directly learn the whole trade-off curve for a
MTL problem in this paper.
As shown in Fig.
2, we want to build a Pareto solution generator to map a preference vector p to its corresponding solution θ p .
If an optimal generator θ p = g ( p | φ ∗ ) is obtained, MTL practitioners can assign their preference via the preference vector p , and directly obtain the corresponding solution θ p with the speciﬁc trade-off among tasks.
With the Pareto solution generator, we can obtain the approximate Pareto set/front as:
[Equation]
where P is the set of all valid preference vectors, g ( p | φ ∗ ) is the Pareto generator with optimal parameters φ ∗ .
Once we have a proper generator g ( p | φ ∗ ) , we can reconstruct the whole approximated set of Pareto stationary solutions ˆ S and the corresponding trade-off curve ˆ F by going through all possible preference vector p from P .
In the rest of this section, we discuss two approaches to deﬁne the form of preference vector p ∈ P and its connection to the corresponding solution θ p =
g ( p | φ ∗ ) .
Preference-Conditioned Linear Scalarization: A simple and straightforward approach is to deﬁne the preference vector p and the corresponding solution θ p via the weighted linear scalarization:
[Equation]
where the preference vector p
= ( p 1 , p 2 , · · · , p m ) is the weight for each task, and the corresponding solution θ p is the minimum of the weighted linear scalarization.
If we further require (cid:80) p i = 1 , the set of all valid preference vector P is an ( m − 1) -dimensional manifold in R m .
Therefore, the valid preference set and the Pareto set are both ( m − 1) -dimensional manifolds.
Although this approach is straightforward, it is not optimal for multiobjective optimization.
Linear scalarization can not ﬁnd any Pareto solution on the non-convex part of the Pareto front (Das &
Dennis, 1997; Boyd & Vandenberghe, 2004).
In other words, unless the problem has a convex Pareto front, the generator deﬁned by linear scalarization cannot cover the whole Pareto set manifold.
Preference-Conditioned Multi-Objective Optimization: To better approximate the manifold of Pareto set, we generalize the idea of decomposition-based multiobjective optimization (Zhang & Li, 2007; Liu et al., 2014) and Pareto MTL (Lin et al., 2019) to connect the preference vector and the corresponding Pareto solution.
To be speciﬁc, we deﬁne the preference p as an m dimensional unit vector in the loss space, and the corresponding solution θ p is the one on the Pareto front which has the smallest angle with p .
The idea is illustrated in in Fig.
With a set of randomly generated unit reference vectors U = { u (1) , · · · , u ( K ) } and the preference vector p , an MTL problem is decomposed into different regions in the loss space.
We call the region closest to p as its preferred region.
The corresponding Pareto solution θ p is the solution that always belongs to the preferred region and on the Pareto front.
Formally, we can deﬁne the corresponding Pareto solution as:
[Equation] [Equation]
) where L ( θ ) is the loss vector and Ω( p , U ) is the constrained preference region conditioned on the preference and reference vectors Ω( p , U ) = { v ∈ R m + | ∠ ( v , p ) ≤ ∠ ( v , u ( j ) ) , ∀ j = 1 , ..., K } .
The constraint is satisﬁed if the loss vector L ( θ ) has the smallest angle with the preference vector p .
The corresponding solution θ p is a restricted Pareto optimal in Ω( p , U ) .
Since we require the preference vectors should be unit vector
|| p
||
= 1 in the m -dimensional space, the set of all valid preference vectors P is an ( m − 1) manifold in R m .
Similar to the linear scalarization case, a preference vector might have more than one corresponding solution, and a Pareto solution can correspond to more than one preference vector.
For simplicity, we assume there is a one-to-one mapping between the preference vector and the corresponding Pareto solution in this paper.
[Figure]
C ONTROLLABLE P ARETO M ULTI -T ASK L EARNING
In this section, we propose a hypernetwork-based deep multi-task neural network framework, along with an efﬁcient end-to-end optimization procedure to solve the MTL problem.
5. H YPERNETWORK - BASED D EEP M ULTI -T ASK N ETWORK
The proposed controllable Pareto multi-task network is shown in Fig. 4(a).
As discussed in the previous section, we use a preference vector to represent a practitioner’s trade-off preference among different tasks.
The preference vector p is the input of the hypernetwork and also deﬁnes the corresponding loss function.
The hypernetwork generates θ p =
g ( p | φ ) as the corresponding parameters for the main MTL network.
The losses vector for the whole network is L ( θ p ) =
L ( g ( p | φ )) and the only trainable parameters to be optimized is the hypernetwork parameters φ .
Practitioners can easily control the MTL network’s parameters, and hence the performances on different tasks in real time, by simply adjusting the preference vector p .
Model Structure and Regularization:
In our proposed model, not matter what preferences p are given, the main MTL network always has a ﬁxed structure, and only the parameters θ p are generated by the hypernetwork.
We consider the main MTL network with hard-shared encoder (Ruder, 2017;
Vandenhende et al., 2020) as shown in Fig. 4(a).
Once the parameters θ p are generated, an input x to the main MTL network will ﬁrst go through the shared encoder, and then all task-speciﬁc heads to obtain the outputs
f i ( x | θ p ) .
Different tasks share the same encoder, and their outputs are still regularized by each other as in a traditional MTL model.
Our proposed model put the cross-tasks regularization on the hypernetwork, where it should generate all parameters for the shared encoder that is good for all tasks conditioned on the given preference.
There is also a regularization effect across different trade-off preferences.
Our goal is to train a single hypernetwork-based model that performs well for all preferences.
In addition, with a proper smoothness assumption, similar preference input to the hypernetwork will have a similar output, which is the generated parameters for the main MTL network as illustrated in Fig. 4(b).
In other words, the generated main MTL network with a speciﬁc preference is regularized by the other networks with similar preferences through the hypernetwork.
This regularization effect might be best explained by the batched preferences update discussed in Appendix A.4.
In this respect, it is similar to von Oswald et al.
(2020) that uses hypernetwork with explicit regularization on separate tasks in the continual learning setting, which is good for knowledge transfer among different preferences.
Scalability: With the model compression ability powered by the hypernetwork (Schmidhuber, 1992;
Ha et al., 2017), our proposed hypernetwork-based model can scale well to large scale problems.
Chunking (Ha et al., 2017; von Oswald et al., 2020) is a commonly used method to reduce the number of parameters for the hypernetwork.
As shown in Fig.
5(a), a hypernetwork can iteratively generate small parts of the main network θ p = [ θ p, 1 , θ p, 2 , · · · , θ p,K ] , with a reasonable model size [Figure] and multiple trainable chunk embedding { e i } Ki =1 , where θ p,k = g ( p , e k | φ ) .
In this way, we can signiﬁcantly compress the model size and make it scalable for large MTL models.
In this paper, we use fully-connected networks as the hypernetwork, and the main MTL neural networks have the same structures as the models used in current MTL literature (Liu et al., 2019; Sener & Koltun, 2018; Vandenhende et al., 2020).
The proposed hypernetwork can scale well for large MTL models and have reasonable latency for real-time adjustment and inference.
The details of the proposed models are provided in the experiments section.
Pretrained Feature Extractor:
Using pretrained feature extractors on a large scale dataset is a powerful way to improve the performance for an MTL model (Ruder, 2017; Vandenhende et al., 2020).
In our proposed method, such a pretrained feature extractor (frozen or unfrozen) can be treated as a structure constraint across all generated solutions as shown in Fig. 5(b).
All generated MTL models have the same parameters for the shared pretrained encoder, while the hypernetwork generates the rest parameters.
This constraint can be treated as regularization on the pretrained dataset.
We analyze the hypernetwork-based model with pretrained feature extractor in the experiment section.
5. E ND - TO -E ND O PTIMIZATION : L EARNING P ARETO G ENERATOR VIA S OLVING MTL
Since we want to control the trade-off preference at the inference time, we need to train a model suitable for all preference vector p in the valid set P but not just a single point.
Suppose we have a probability distribution on all possible preference vector p ∼ P p , a general goal would be min φ E p ∼ P p L ( g ( p | φ )) .
It is hard to optimize the parameters within the expectation directly.
We use Monte Carlo method to sample the preference vector, and use the stochastic gradient descent algorithm to train the deep neural network.
At each step, we can sample one preference vector p and optimize the loss function:
[Equation]
[Equation] At each iteration t , if we have a valid gradient direction d t for the loss L ( g ( p | φ t )) , we can simply update the parameters with gradient descent φ t +1 =
φ t − η d t .
For the preference-conditioned linear scalarization case as in problem (4), the calculation of d t is straightforward:
[Equation]
For the preference-conditioned multiobjective optimization problem (5), a valid descent direction should simultaneously reduce all losses and activated constraints.
One valid descent direction can be written as a linear combination of all tasks with dynamic weight α i ( t ) :
[Equation]
where the coefﬁcients α i ( t ) is depended on both the loss functions
L i ( g ( p | φ t )) and the constraints
G j ( θ | p , U ) .
We give the detailed derivation in the Appendix A.2 due to page limit.
[Figure]
6: end
7: Output φ
The algorithm framework is shown in Algorithm 1 .
We use a simple uniform distribution on all valid preferences and references vector in this paper.
At each iteration, we calculate a valid gradient direction and update the hypernetwork parameters.
The proposed algorithm simultaneously optimizes the hypernetwork for all valid preference vectors that represent different valid trade-offs among all tasks.
As shown in Fig.
6, it continually learns the trade-off curve during the optimization process.
We obtain a Pareto generator at the end and can directly generate a solution θ p =
g ( p | φ T ) from any valid preference vector p .
By taking all preference vectors as input, we can construct the whole trade-off curve.
We give a discussion on the convergence-diversity analysis of the proposed algorithm in the Appendix Section B
S YNTHETIC M ULTI -O BJECTIVE O PTIMIZATION P ROBLEM
[Figure]
proposed controllable
Pareto MTL method can successfully generate the whole Pareto front from the preference vectors.
In contrast, the linear scalarization method is only accurate for Pareto solutions near the endpoints.
In the previous section, we propose to use a hypernetwork to generate the parameters for the main deep multi-task network based on a preference vector.
The network parameters are in a highdimensional decision space, and the optimization landscape would be complicated.
To better analyze the behavior and convergence performance of the proposed algorithm, we ﬁrst use it to generate the
Pareto solutions for a low-dimensional multiobjective optimization problem.
The experimental results are shown in Fig.
This synthetic problem has a sin-curve-like Pareto set in the solution space, and its corresponding Pareto front is a concave curve in the objective space.
The detailed deﬁnition is given in the appendix.
Our proposed controllable Pareto MTL method can cover and reconstruct the whole manifold of Pareto front from the preference vectors, while the traditional methods can only ﬁnd a ﬁnite point set approximation.
It should be noticed that the simple linear scalarization method has poor performance for the problem with concave Pareto front.
E XPERIMENTS
[Figure] [Figure]
In this section, we validate the performance of the proposed controllable Pareto MTL to generate the trade-off curve for different MTL problems.
We compare it with the following MTL algorithms: 1) Linear Scalarization: simple linear combination of different tasks with ﬁxed weights; 2)Uncertainty (Kendall et al., 2018): adaptive weight assignments with balanced uncertainty; 3)DWA (Liu et al., 2019): dynamic weight average for the losses; 4) MGDA (Sener & Koltun, 2018): ﬁnds one Pareto solution; 5) Pareto MTL (Lin et al., 2019): generates a set of wildly distributed Pareto solutions; and 6) Single Task: the single task baseline.
We conduct experiments on the following widely-used MTL problems: 1) MultiMNIST (Sabour et al., 2017):
This problem is to classify two digits on one image simultaneously, where the digits are on the top left and bottom right respectively.
2) CityScapes (Cordts et al., 2016):
This dataset has street-view RGB images, and involves two tasks to be solved, which are pixel-wise semantic segmentation and depth estimation.
NYUv2 (Silberman et al., 2012):
This dataset is for indoor scene understanding with two tasks: a 13 -class semantic segmentation and indoor depth estimation.
4) CIFAR-100 with 20 Tasks: Follow a similar setting in (Rosenbaum et al., 2018), we split the original CIFAR-100 dataset (Krizhevsky & Hinton, 2009) into 20 ﬁve-class classiﬁcation tasks.
Details of these problems can be found in the Appendix D.
Result Analysis: The experimental results are shown in Fig.
8 where all models are trained from scratch.
Our proposed algorithm can learn the entire trade-off curve for different MTL problems.
In addition, the proposed preference-conditioned multiobjective optimization method can always ﬁnd a better trade-off curve that dominates the curve found by simple linear scalarization.
These results validate the effectiveness of the proposed model and the end-to-end optimization method.
For all experiments, the single-task model is a strong baseline in performance.
However, it cannot dominate most of the approximated Pareto front learned by our model.
Our models provide diverse
[Table]
optimal trade-offs among tasks (e.g., in the upper left and lower right area) for different problems.
In addition, the inference time would be a concern for the single task baseline ( m full models).
The experimental result on the 20 -tasks CIFAR100 MTL problem is shown in Fig.
Our proposed model can achieve the best overall performance among different tasks by making a real-time trade-off adjustment.
It also outperforms the balanced hypernet approach, which uses the same hypernetwork-based model to optimize all tasks simultaneously.
[Figure]
Scalability: The models we use in the experiments are summarized in Table.1.
As discussed in the earlier sections, we build the hypernetwork such that the proposed models have a comparable number of parameters with the traditional MTL model.
Given the current works (Lin et al., 2019; Mahapatra & Rajan, 2020; Ma et al., 2020) need to train and store multiple MTL models to approximate the Pareto front, our proposed model is much more parameter-efﬁcient while learning the whole trade-off curve.
In this sense, it is more scalable than the current methods.
Inference and Latency: The GPU latencies for all models are reported in Table.1.
Similar to other hypernetwork-based models (Ha et al., 2017; von Oswald et al., 2020), we do not need to solve any optimization problem at the inference time.
For inference, the hypernetwork takes the preference vector as input, and directly generate the parameters for the main MTL network.
It leads to an affordable overhead on the GPU latency, which is suitable for real-time preference adjustment.
Experiment with Pretrained Backbone: We conduct an experiment on the NYUv2 problem with a large-scale ResNet-50 backbone following the experiment settings in the current MTL survey paper (Vandenhende et al., 2020).
Each model has a ResNet-50 backbone with weights pretrained on ImageNet, and task-speciﬁc heads with ASPP module (Chen et al., 2018a).
The results are shown in Fig.10, where the results for MTL models are from Vandenhende et al.
(2020) with best ﬁnetuned conﬁgurations.
Our proposed model also uses the pretrained ResNet-50 backbone, but generates the encoder’s last few layers and all task-speciﬁc heads.
Our proposed method can learn the trade-off curves with comparable performance, while supporting real-time trade-off adjustment.
C ONCLUSION
In this paper, we proposed a novel controllable Pareto multi-task learning framework for solving
MTL problems.
With a hypernetwork-based Pareto solution generator, our method can directly learn the whole trade-off curve for all tasks with a single model.
It allows practitioners to easily make real-time trade-off adjustment among tasks at the inference time.
Experimental results on various MTL applications demonstrated the usefulness and efﬁciency of the proposed method.
A PPENDIX
As mentioned in the main paper, we provide more discussion and analysis in this appendix, which can be summarized as follows:
• Pareto Generator and Multiobjective Gradient Descent: We discuss the idea of Pareto solution generator from the view of multiobjective optimization in Section A. We also give the details of preference-conditioned multiobjective gradient descent for generator training.
• Convergence-Diversity Analysis for Multiobjective Optimization: In Section B, we provide a brief analysis on the convergence-diversity trade-off for multiobjective optimization.
We discuss the convergence from multiple points to the manifold learning.
• Scalability for Hypernetwork-based Pareto Generator: We discuss the scalability for our proposed hypernetwork-based Pareto generator, and introduce two easy-to-use approaches to improve scalability in Section C. • Experimental Setting: We provide detailed experimental setting in Section ?? .
A P ARETO G ENERATOR AND M ULTIOBJECTIVE G RADIENT D ESCENT
This work follows a recent line of research on introducing multiobjective optimization into multitask learning (Sener & Koltun, 2018;
Lin et al., 2019).
The previous works simply bridge the two ﬁelds, and use or combine existing multiobjective optimization methods to solve MTL problems.
In contrast, we ﬁnd that the current multiobjective optimization methods have shortcomings and are insufﬁcient for many MTL applications.
This work proposes a novel Pareto solution generator for multiobjective optimization, which is useful for solving MTL problems.
The proposed method to learn the whole Pareto set manifold is a new contribution and could be useful for both communities.
In this section, we discuss the idea of Pareto solution generator from the view of multiobjective optimization, and show how this algorithm can be used to solve MTL problems.
A. P ARETO S OLUTION G ENERATOR
solution generator directly maps a preference vector to the corresponding Pareto solution in the decision space.
By adjusting the preference vector, a practitioner can control the generated Pareto solution to explore the whole manifold of the Pareto set with a single model in real time.
In the viewpoint of multiobjective optimization, we provide a controllable Pareto solution generator, instead of a set of ﬁnite solutions, to approximate the manifold of Pareto set:
Traditional:
[Equation]
Pareto Generator:
[Equation]
The goal of most traditional multiobjective optimization algorithms is to ﬁnd a set of ﬁnite solutions to approximate the Pareto set.
However, under mild condition, since the Pareto set is a manifold, no set of ﬁnite solutions would be a perfect estimator.
This approximation also restricts the information provided to the practitioners to make a decision.
If no solution in the approximation set satisﬁes the speciﬁc trade-off preference, the practitioners need to rerun the algorithm to obtain other solutions.
[Figure]
As shown in Fig.
11, our proposed Pareto solution generator directly maps a speciﬁc preference vector to the corresponding Pareto solution.
In the ideal case, the generator can cover the whole ( m − 1) dimensional Pareto set manifold by exploring the suitable preference set.
The Pareto solution generator also provides more information to the practitioners, and allow them to make ﬂexible realtime control over the Pareto solutions.
The practitioners can adjust the preference vector to obtain a suitable Pareto solution, or adaptively set the trade-off preference in real time to make a dynamic adjustment, which could be crucial for many real-world applications.
A. P REFERENCE -C ONDITIONED M ULTIOBJECTIVE G RADIENT D ESCENT
The Pareto solution generator generates a solution θ p =
g ( p | φ ) from the preference vector p .
Under ideal condition, if we take all valid preference vectors p ∈ P (which is an ( m − 1) -dimensional manifold) as input, the output would be another ( m − 1) -dimensional manifold.
At the end of the algorithm, we hope the generated manifold is close to the manifold of true Pareto set.
In the proposed algorithm, we randomly sample a preference vector p at each iteration to update the generator parameters φ .
As illustrated in Fig.
12, it continually learns and improves the current manifold around the preference vector at each iteration.
It should be noticed that the sampled preference vector and all reference vectors are different at each iteration, so as the decomposition.
the valid gradient direction should reduce all losses and activated constraints for the generated solution.
As mentioned in the Algorithm 1 in the main paper, we use simple gradient descent to update the Pareto solution generator at each iteration.
For the case of linear scalarization, calculating the gradient direction is straightforward.
In this subsection, we discuss how to obtain a valid gradient direction for the preference-conditioned multiobjective optimization.
Similar to the previous work (Sener & Koltun, 2018; Lin et al., 2019), we use multiobjective gradient descent (Fliege & Svaiter, 2000; D´esid´eri, 2012) to solve the MTL problem.
However, in our algorithm, the optimization parameter is φ for the Pareto solution generator rather than θ for a single solution.
A simple illustration of the multiobjective gradient descent idea for a problem with two tasks is shown in Fig.
At each iteration, the algorithm samples a preference vector p and obtains its current corresponding solution θ p .
A valid gradient direction should reduce all the losses and guide the generated solution θ p =
g ( p | φ ) toward the preference region Ω( p , U ) around the preference vector p .
At iteration t , for a given preference vector p , the preference-conditioned multiobjective problem with the Pareto solution generator can be written as:
[Equation]
Since the parameter θ p is generated by the Pareto solution generator g ( p | φ t ) , the generator’s parameter φ t is the only trainable parameters to be optimized.
What we need is to ﬁnd a valid gradient direction d t to reduce all the losses
L i ( g ( p | φ t )) and activated constraints G j ( g ( p | φ t ) | p , U ) with respect to φ t .
We follow the methods proposed in (Fliege & Svaiter, 2000; Gebken et al., 2017), and calculate a valid descent direction d t by solving the optimization problem:
[Equation]
where d t is the obtained gradient direction, α is an auxiliary parameter for optimization, and I ( θ ) =
{ j ∈ I |G
j ( g ( p | φ t )) ≥ 0 } is the index set of all activated constraints.
By solving the above problem, the obtained direction d t and parameter α t will satisfy the following lemma (Gebken et al., 2017):
Lemma 1: Let ( d t , α t ) be the solution of problem (12), we either have:
A non-zero d t and
α t with
[Equation]
or d t = 0 ∈ R n , α t = 0 , and θ p =
g ( p | φ t ) is local Pareto critical restricted on Ω( p , U ) .
In case 1, we obtain a valid descent direction d t (cid:54) = 0 which has negative inner products with all ∇ φ L i ( g ( p | φ t )) and ∇ φ G j ( g ( p | φ t )) for j ∈ I ( φ t ) .
With a suitable step size η , we can update φ t +1 =
φ t + η d t to reduce all losses and activated constraints.
In case 2, we cannot ﬁnd any nonzero valid descent direction, and obtain d t = 0 .
In other words, there is no valid descent direction to simultaneously reduce all losses and activated constraints.
Improving the performance for one task would deteriorate the other(s) or violate some constraints.
Therefore, the current solution θ p = g ( p | φ t ) is a local restricted Pareto optimal solution on Ω( p , U ) .
A. A DAPTIVE L INEAR S CALARIZATION
As mentioned in the main paper, we can rewrite the gradient direction d t as a dynamic linear combination with the gradient of all losses ∇ φ L i ( g ( p | φ t )) .
Similar to the approach in (Fliege & Svaiter, 2000; Lin et al., 2019), we reformulate the problem (12) in its dual form:
[Equation]
where d t = (cid:80) mi
=
λ i ∇ φ t L i ( g ( p | φ t ))
+ (cid:80)
j ∈ I ( φ t ) β i ∇ φ t G j ( g ( p | φ t )) is the obtained valid gradient direction, λ i and β i are the Lagrange multipliers for the linear inequality constraints in problem (12).
Based on the deﬁnition in problem (11), a constraint G j ( g ( p | φ t )) can be rewritten as a linear combination of all losses:
[Equation]
[Figure]
The gradient of the constraint is also a linear combination of those for the losses:
[Equation]
Therefore, we can rewrite the valid descent direction as a linear combination of the gradients for all tasks with dynamic weight α i ( t ) :
[Equation]
where λ i and β j are obtained by solving the dual problem (14).
It should be noticed that the primal problem (12) directly optimizes the gradient direction d t , which could be in millions of dimensions for training a deep neural network.
In contrast, the dimension of dual problem (14) is the number of all tasks and activated constraints (up to a few dozens), which is much smaller than the primal problem.
Therefore, in practice, we obtain the valid gradient direction d t by solving the dual problem (14) instead of the primal problem (12).
We use the Frank-Wolfe algorithm
(Jaggi, 2013) to solve the problem (14) as in the previous work (Sener & Koltun, 2018; Lin et al., 2019).
We use simple uniform distribution to sample both the unit preference vector p and unit reference vectors U in this paper.
The number of preference vector is 1 in the previous discussion, and the number of reference vectors is a hyperparameter, which we set it as 3 in this paper.
In the next subsection, we will discuss how to use more than one preference vector at each iteration to update the Pareto solution generator.
A. B ATCHED P REFERENCES U PDATE
To obtain an optimal Pareto generator g ( p | φ ∗ ) , we need to ﬁnd:
[Equation]
In the main paper, we sample one preference vector at each iteration to calculate a valid direction d t to update the Pareto generator.
A simple and straightforward extension is to sample multiple preference vectors to update the Pareto solution generator, as shown in Fig.
At each iteration, we simultaneously sample and optimize multiple subproblems with respect to the
Pareto generator:
[Equation]
where p 1 , p 2 , · · · , p K are K randomly sampled preference vectors and each L ( g ( p k | φ )) is a multiobjective optimization problem.
Therefore, we now have a hierarchical multiobjective optimization problem.
If we do not have a speciﬁc preference among the sampled preference vectors, the above problem can be expanded as a problem with Km objectives:
[Equation]
For the linear scalarization case, the calculation of valid gradient direction at each iteration t is straightforward:
[Equation]
where we assume all sampled subproblems are equally important
[Equation]
It is more interesting to deal with the preference-conditioned multiobjective optimization problem.
For each preference vector p k , suppose we use the rest preference vectors as its reference vectors, the obtained preference-conditioned multiobjective optimization problem would be:
[Equation]
There are ( K − 1) constraints for each preference vector, hence total K ( K − 1) constraints for the preference-conditioned multiobjective problem, although some of them could be inactivated.
Similar to the single point case, we can also calculate the valid gradient direction in the form of adaptive linear combination as:
[Equation]
23) where the adaptive weight β
i ( t ) depends on all loss functions
L i ( g ( p k | φ t )) and activated constraints G kj ( g ( p k | φ )) .
[Figure]
B C ONVERGENCE -D IVERSITY A NALYSIS
In the previous section, we have discussed how to ﬁnd a valid gradient direction to update the Pareto solution generator by sampling one or a set of preference vectors at each iteration.
For a preference vector p , if no valid descent direction can be found, the generated solution θ p converges to a local restricted Pareto optimal point.
It would be much more complicated to study the convergence of the whole manifold, especially when we use a neural network to approximate the unknown Pareto front.
Instead of giving proof and theorem, we give a general discussion on the convergence and diversity analysis from the view of multiobjective optimization.
We hope this discussion can lead to a better understanding of our proposed algorithm, and could be useful for potential future work on learning the whole Pareto front.
B. S OLVING MOP WITH F INITE S OLUTIONS : C ONVERGENCE AND D IVERSITY
For a single objective optimization problem, we care about the convergence of a single point to the local or global optimum.
For a multiobjective optimization problem, the traditional algorithms usually focus on ﬁnding one Pareto solution or a set of ﬁnite solutions to approximate the Pareto front.
Since the Pareto front is a ( m − 1) manifold in the objective space, no set of ﬁnite solutions can perfectly represent the whole Pareto front.
With a ﬁxed number of approximate solutions, there is a convergence-diversity dilemma for the multiobjective optimization algorithms.
The current convergence analysis is still mainly on the single point-based algorithm (Fliege et al., 2019).
As shown in Fig.
15, a good approximation for the Pareto front should have a good balance between the convergence and diversity.
However, there is no unique best way to deﬁne the optimal balance between convergence and diversity for a multiobjective optimization problem.
In fact, the deﬁnition of diversity is also not unique, and different methods have their pros and cons for different problems.
Different criteria and indicators have been proposed to measure the quality of a set of approximated solutions (Deb, 2001; Miettinen, 2012).
They have different concerns and balances between the convergence and diversity, but there is no single best one for all problems.
How to fairly evaluate the quality for a set of approximated solutions (e.g., the level it ”converges” to the whole Pareto front) is still an open question in the multiobjective optimization community.
B. P ARETO S OLUTION G ENERATOR : C ONVERGENCE
Different from the ﬁnite Pareto solutions approximation, our work proposes to use a Pareto solution generator to learn the whole Pareto front.
In the optimal case, by taking all valid preference vectors as input (which is an ( m − 1) -manifold), it should generate all solutions on the Pareto set in the solution space, with the corresponding Pareto front in the objective space (we assume one-to-one mapping for all cases).
As shown in Fig.
15(d), for an optimal generator, all generated solutions should be on the Pareto front, and the Pareto front should be well covered by the generated solutions.
[Figure]
For a ﬁxed preference vector p , the valid gradient d t = 0 means the corresponding solution θ p =
g ( p | φ t ) is on the Pareto set and hence it is a Pareto solution.
For the Pareto solution generator, if we obtain optimal parameters φ ∗ , the gradient direction d should be 0 for all preference vector p in P .
In this case, all solutions generated from valid preference vectors are Pareto solution as in Fig. 16(a).
If we only want to obtain a single Pareto solution ( K = 1 ), we can adapt the current theory results on single-point convergence to analyze the generator’s convergence performance (Fliege et al., 2019).
The case could become difﬁcult and challenging once we have inﬁnite preferences.
As discussed in the main paper and previous section, the goal to be optimized for a Pareto generator is:
[Equation]
If the number of preferences is ﬁnite (e.g., P p is a discrete distribution), we can use the batched preference method (A.4) to update the generator at each iteration.
If we have a large number of even inﬁnite preferences (e.g., P p is a continuous distribution), the single and batch method can be seemed as a sampling approximation to optimize the expectation, which is similar to SGD or batched SGD against full gradient descent.
B. P ARETO S OLUTION G ENERATOR : D IVERSITY
preference vector p at different iterations
The previous subsection discusses the convergence issue.
However, as shown in Fig 16(b), only requiring all preferences have gradient direction d t = 0 is not sufﬁcient to deﬁne the optimality of the Pareto generator, since it can not avoid convergence to a narrow local region on the
Pareto front.
In our proposed algorithm, we connect each preference vector to its corresponding Pareto solution via the Pareto solution generator g ( p | φ t ) .
During the optimization process, we also require the generated solution should be close to the region around the preference vector.
At each iteration, we randomly sample the preference vector p and a set of reference vectors U to decompose the loss space into different regions.
A given preference vector p might have different preference regions at different iterations.
However, all the corresponding regions should be around the preference vector, and their overlapped region should be very close to the preference vector as in Fig.
As shown in Fig.
16(c), if the preference vectors are widely distributed in the loss space, and the generated solutions are on the Pareto front and close to their own preference vectors, the proposed algorithm can provide a good approximation to the ground-truth Pareto set and Pareto front.
The remaining question is how to guarantee the preferences vectors can represent and cover the whole Pareto front.
In our proposed algorithm, the preference vectors are sampled from the distribution P p at each iteration.
The remaining question is how to properly set the distribution function to cover the ground-truth but unknown Pareto front well.
If we have enough knowledge about the given MTL problem, we might deﬁne a ﬁxed and well-behaved distribution P p before the optimization process.
However, in real-world applications, the more realistic case is that we know nothing about the Pareto front before actually training the MTL model.
Some recent works have studied the relation between the decomposition method and the convergence-diversity indicator (e.g., hypervolume) (Golovin et al., 2020), which might be useful to adaptively adjust the preference distribution
P p during the optimization process.
[Figure]
C S CALABILITY I MPROVEMENT
In this paper, we propose to learn the whole Pareto front for an MTL problem, and generate corresponding Pareto solutions based on given preferences in real time.
Since both the preference vector and the Pareto front are ( m − 1) -dimensional manifolds, if we assume the one-to-one mapping does exist, there could be suitable parametric functions to connect them.
However, since we are training large scale neural networks, it still has concerns on learning and generating the network parameters.
The generated network parameters are in high dimensional parameter space, although the targeted
Pareto set is a low-dimensional manifold.
We believe the hypernetwork can work for large-scale real-world applications for the following reasons: (i)
Recent studies make training hypernetwork more robust for large-scale networks (Chang et al., 2020).
(ii)
The idea of hypernetwork is originally designed for model compression (Schmidhuber, 1992; Stanley et al., 2009; Bertinetto et al., 2016; Ha et al., 2017).
Many methods can be used to reduce the size of hypernetwork-based model.
A chunking method (Ha et al., 2017; von
Oswald et al., 2020) is shown in Fig. 18(a).
With different low-dimensional trainable embeddings, one can use a small-size hypernetwork to generate a larger network’s parameters.
(iii)
The broader multiplicative interaction methods (Jayakumar et al., 2020) might lead to further improvement.
This paper does not focus on designing sophisticated MTL networks, but the following two easy-touse approaches would be useful to improve the scalability of the proposed Pareto solution generator:
Parameter Sharing:
As shown in Fig.
18(b), we can force different trade-off solutions to share the same low-level feature extractor (part of the encoder).
In other words, the hypernetwork only generates part of the main MTL network, and the rest parameters are shared among different Pareto solutions.
Now the obtained manifold would be a restrict Pareto front, with constraints on the sharing parameters.
This form could also be beneﬁcial to knowledge transfer.
For example, we can use a pretrained feature extractor (frozen or unfrozen) as the shared parameters, and then build a hypernetwork-based structure on top to generate the Pareto front.
The hypernetwork-based model itself is a dynamic feature extractor, can be used to transfer knowledge to other models.
More sophisticated parameter sharing methods could be studied in the future.
Preference Embedding: The original input for the hypernetwork is the preference vector, which is typically in a low-dimensional space.
One straightforward way to expand the input dimension is to use preference embedding.
As shown in Fig.
18(c), if we assign a trainable q -dimensional embedding e i for each task, then the input to the hypernetwork would be (cid:80) mi p i e i which is also a q -dimensional vector.
We can set a larger q (e.g., q = 100 ) for problem with small m (e.g., m = 2 ).
If we set each e i as a one-hot (untrainable) m -dimensional unit vector, the input will reduce to the original preference vector p .
In the simples case (cid:80) mi p
i e i , we are doing learnable linear interpolation among the preference embedding e i , which can be trained during the optimization process.
Then the highly nonlinear hypernetwork maps the learned embedding into the main MTL network’s parameters.
We can also use other powerful nonlinear embedding methods, such as the kernel method and multiplicative interaction method, to further improve this approach.
D E XPERIMENTAL S ETTING
Synthetic Example: The synthetic example we use in section 6 is deﬁned as:
[Equation]
We set n = 10 and use a simple two-layer MLP network to generate the Pareto solutions based on the preference vectors.
MultiMNIST: In this problem, the goal is to classify two overlapped items in an image.
The size of the original MNIST image is 28 × 28 .
We randomly choose two items from a dataset, and move one item to the upper-left and the other one to the bottom right with up to 4 pixels.
Therefore, the input image is 36 × 36 .
Similar to the previous work (Sener & Koltun, 2018; Lin et al., 2019), we use a LeNet-based network with task-speciﬁc FC layers as the MTL model.
For the hypernetworkbased model, we use simple MLP as the hypernetwork.
For all methods, the optimizer is Adam with learning rate lr = 3 e − 4 , the batch size is 256 , and the number of epochs is 200 .
CityScapes: This problem has two tasks to be learned together, which is the semantic segmentation and depth estimation.
We follow the setting used in (Liu et al., 2019), and resize all images into 128 × 256 .
For the semantic segmentation, the model predicts the 7 coarser labels for each pixel.
We use the L1 loss for the depth estimation.
We report the experimental results on the Cityscapes validation set.
We use the MTL network proposed in (Liu et al., 2019), which has SegNet (Badrinarayanan et al., 2017) as the shared representation encoder, and two task-speciﬁc lightweight convolution layers.
For all experiments, we use Adam with learning rate lr = 3 e − 4 as the optimizer, and the batch size is 12 .
We train the model from scratch with 200 epochs.
NYUv2:
This problem has a 13-class segmentation task and a depth estimation task for indoor scenes.
We resize all images into 288 × 384 .
We use a similar MTL network and hyperparameter setting as for the CityScapes problem, except the batch size is 3 in this problem.
CIFAR100 with 20 Tasks:
To validate the algorithm performance on MTL problem with many tasks, we split the CIFAR-100 dataset (Krizhevsky & Hinton, 2009) into 20 tasks, where each task is a 5 -class classiﬁcation problem.
Similar setting has been used in MTL learning (Rosenbaum et al., 2018) and continual learning (von Oswald et al., 2020).
The MTL neural network has four convolution layers as the shared architecture, and 20 task-speciﬁc FC layers.
The optimizer is Adam with learning rate lr = 3 e − 4 , and the batch size is 128 .
We report the test accuracy for all 20 tasks.