P ARETO M ANIFOLD L EARNING : T ACKLING MULTIPLE TASKS VIA ENSEMBLES OF SINGLE - TASK MODELS
Anonymous authors
Paper under double-blind review
A BSTRACT [All reviewers]: We have added ”notes” such as this one to make the changes more vis- ible. For minor things such as slightly changing plot size, we do not write anything.
In Multi-Task Learning, tasks may compete and limit the performance achievedon each other rather than guiding the optimization trajectory to a common solution, superior to its single-task counterparts.
There is often not a single solutionthat is optimal for all tasks, leading practitioners to balance tradeoffs betweentasks’ performance, and to resort to optimality in the Pareto sense.
Current MultiTask Learning methodologies either completely neglect this aspect of functionaldiversity, and produce one solution in the Pareto Front predeﬁned by their optimization schemes, or produce diverse but discrete solutions, each requiring aseparate training run.
In this paper, we conjecture that there exist Pareto Subspaces, i.e., weight subspaces where multiple optimal functional solutions lie.
Wepropose Pareto Manifold Learning , an ensembling method in weight space that isable to discover such a parameterization and produces a continuous Pareto Frontin a single training run, allowing practitioners to modulate the performance oneach task during inference on the ﬂy.
We validate the proposed method on a diverse set of multi-task learning benchmarks ranging from image classiﬁcation totabular datasets and scene understanding, and show that Pareto Manifold Learningoutperforms state-of-the-art algorithms.
I NTRODUCTION
In Multi-Task Learning (MTL), multiple tasks are learned concurrently within a single model, striving towards infusing inductive bias that will help outperform the single-task baselines.
Apart fromthe promise of superior performance and some theoretical beneﬁts (Ruder, 2017), such as generalization properties for the learned representation, modeling multiple tasks jointly has practical beneﬁtsas well, e.g., lower inference times and memory requirements.
However, building machine learningmodels presents a multifaceted host of decisions for multiple and often competing objectives, suchas model complexity, runtime and generalization.
Conﬂicts arise since optimizing for one metric often leads to the deterioration of other(s).
A single solution satisfying optimally all objectives rarelyexists and practitioners must balance the inherent trade-offs.
In contrary to single-task learning, where one metric governs the comparison between methods (e.g.,top-1 accuracy in ImageNet ), multiple models can be optimal in Multi-Task Learning; e.g., model
X yields superior performance on task A compared to model Y, but the reverse holds true for task B ;thus, there is not a single better model among the two.
This notion of tradeoffs is formally deﬁnedas Pareto optimality .
Intuitively, improvement on an individual task performance can come only atthe expense of another task.
However, there exists no framework addressing the need for efﬁcientconstruction of the Pareto Front, i.e., the set of all Pareto optimal solutions.
Recent methods in Multi-Task Learning casted the problem in the lens of multi-objective optimization and introduced the concept of Pareto optimality, resulting in different mechanisms for computing the descent direction for the shared parameters.
Speciﬁcally, Sener & Koltun (2018) produce asingle solution that lies on the Pareto Front.
As an optimization scheme, however, it is biased towards the task with the smallest gradient magnitude, as argued in Liu et al.
Lin et al.
(2019)expand this idea and, by imposing additional constraints on the objective space to produce multiplesolutions on the Pareto Front, each corresponding to a different user-speciﬁed tradeoff.
Finally, thework by Ma et al.
(2020) proposes an orthogonal approach that can be applied after training andstarts with a discrete solution set and produces a continuous set (in weight space) around each so- lution, while the overall Pareto Front is continuous only in objective space as the union of the local(weight-space continuous)
Pareto Fronts.
As a consequence, the memory requirements grow linearlywith the number of models stored. Navon et al. (2021); Lin et al.
(2021) use hypernetworks to produce a Pareto Front in a single training run, but this approach has limited scalability and introduces additional choices
[All reviewers]: Added a short clariﬁcation about related work.
More details are also provided in ”Related work” section. In this paper, we conjecture that we can actually produce a subspace with multiple Pareto stationarypoints in the Multi-Task Learning setting with the hypothesis that local optima (produced bydifferent runs or sharing training steps) can be found in close proximity and are connected bysimple paths.
This is motivated by the recent advancements in single task machine learning thathave explored the geometry of the loss landscape and shown experimentally that local optima areconnected by simple paths, even linear ones in some cases (Wortsman et al., 2021; Garipov et al.
,2018; Frankle et al., 2020; Draxler et al., 2018).
We assume that, when the problem has multipleobjectives, it acquires a new dimension relating to the number of tasks.
Concretely, there aremultiple loss landscapes and a solution that satisﬁes users’ performance requirements must lie inthe intersection of low loss valleys (for all tasks).
Building upon our conjecture, we develop a novel method, Pareto Manifold Learning , which casts
Multi-Task problems as learning an ensemble of single-task predictors by interpolating among (ensemble) members during training.
By operating in the convex hull of the members’ weight space,each single-task model infuses and beneﬁts from representational knowledge to and from the othermembers.
During training, the losses are weighted in tandem with the interpolation, i.e., a monotonic relationship is imposed between the degree of a single-task predictor participation and theweight of the corresponding task loss.
Consequently, the ensemble as a whole engenders a (weight)subspace that explicitly encodes tradeoffs and results in a continuous parameterization of the Pareto
Front.
We identify challenges in guiding the ensemble to such subspaces, designated Pareto subspaces , and propose solutions regarding balancing the loss contributions, and regularizing the Paretoproperties of the subspaces and adapting the interpolation sampling distribution.
Experimental results validate that the proposed method is able to discover Pareto Subspaces , and outperforms baselines on multiple benchmarks.
Our training scheme offers two main advantages.
First,enforcing low loss for all tasks on a linear subspace implicitly penalizes curvature, which has beenlinked to generalization (Chaudhari et al., 2017), beneﬁtting all tasks’ performance.
Second, the algorithm produces a subspace of Pareto Optimal solutions, rather than a single model, enabling practitioners to handpick during inference the solution that offers the tradeoff that best suits their needs.
R ELATED W ORK
Multi-Task Learning.
Learning multiple tasks in the Deep Learning setting (Ruder, 2017; Crawshaw, 2020) is usually approached by architectural methodologies (Misra et al., 2016; Ruder et al. ,2019), where the architectural modules are combined in several layers to govern the joint representation learning, or optimization approaches (Cipolla et al., 2018;
Chen et al., 2018), where thearchitecture is standardized to be an encoder-decoder(s), for learning the joint and task-speciﬁc rep- resentations, respectively, and the focus shifts to the descent direction for the shared parameters.
We focus on the more general track of optimization methodologies ﬁxing the architectural structure to Shared-Bottom (Caruana, 1997).
The various approaches focus on ﬁnding a suitable descentdirection for the shared parameters.
The optimization methods can be broadly categorized into lossbalancing and gradient-balancing (Liu et al., 2020).
For the former, the goal is to compute anappropriate weighting scheme for the losses, e.g., the losses can be weighted via task-dependent homoscedastic uncertainty (Cipolla et al., 2018), by enforcing task gradient magnitudes to have closenorms (Chen et al., 2018).
[Table]
The latter class of methodologies manipulate the gradients so that theysatisfy certain conditions; projecting the gradient of a (random) task on the normal plane of anotherso that gradient conﬂict is avoided (Yu et al., 2020), enforcing the common descent direction tohave equal projections for all task gradients (Liu et al., 2020), casting the gradient combination as abargaining game (Navon et al., 2022).
Multi-Task Learning for Pareto Optimality.
The authors in (Sener & Koltun, 2018) were the ﬁrstto view the search for a common descent direction under the Pareto optimality prism and employthe Multiple Gradient Descent Algorithm (MGDA) (D´esid´eri, 2012) in the Deep Learning context.
However.
MGDA did not account for task preferences and the solutions yielded for various initializations in a synthetic example resulted in similar points in the Pareto Front (Lin et al., 2019).
Bysolving a slightly different formulation of the multi-objective problem, they are able to systematically introduce task trade-offs and produce a discrete Pareto Front.
However, this approach requiresas many training runs as the stated preference combinations and the optimization process for eachtraining step of each run introduces a non-negligible overhead.
The work in (Ma et al., 2020) proposes an orthogonal approach for Pareto stationary points; after a model is ﬁtted with any Multi-Task Learning method and has converged to a point (seed) in parameter space, a separate phase seeksother Pareto stationary points in the vicinity of the seed.
The convex hull of these points is guaranteed to lie in the Pareto Front.
But training still needs to occur for every seed point, the separatephase overhead grows linearly with the number of additional models, and the Pareto Front is notcontinuous across seed points in parameter space . Navon et al.
(2021) and Lin et al. (2021) employhypernetworks to continuously approximate the Pareto Front in a single run, which introduces additional design choices. Ruchte & Grabocka (2021) address the scalability issues of hypernetworksby augmenting the feature space with the preference vector. Raychaudhuri et al.
(2022) employ asecond hypernetwork to also modulate the architecture of the target network addressing. [All reviewers]: Added prior work.
Ensemble Learning and Mode Connectivity.
Apart from Multi-Task Learning, our algorithm ismethodologically tied to prior work in the geometry of the neural network optimization landscapes.
The authors in (Garipov et al., 2018; Draxler et al., 2018) independently and concurrently showedthat for two local optima θ ∗ 1 , θ ∗ 2 produced by separate training runs (but same initializations) thereexist nonlinear paths, deﬁned as connectors by Wortsman et al.
(2021), where the loss remains low.
The connectivity paths can be extended to include linear in the case of the training runs sharing somepart of the optimization trajectory (Frankle et al., 2020).
These ﬁndings can be leveraged to train aneural network subspace by enforcing linear connectivity among the subspace endpoints (Wortsmanet al., 2021). Appendix J discusses more related work regarding ensemble learning and ﬂat minima.
P ROBLEM F ORMULATION
Notation.
We use bold font for vectors x , capital bold for matrices X and regular font for scalarsx .
T is the number of tasks and m is the number of ensemble members.
Each task t ∈ [ T ] has a loss
L t .
The overall multi-task loss is L =
[ L 1 , . . .
, L T ] (cid:62) .
w ∈ ∆ T ⊂ R T is the weighting scheme forthe tasks, i.e., the overall loss is calculated as L = w (cid:62) L
= (cid:80)
Tt =1 α t L t .
Each member k ∈ [ m ] isassociated with parameters θ k ∈ R N and weighting w ∈ ∆ T .
Preliminaries.
Our goal lies in solving an unconstrained vector optimization problem of minimizing L ( y , ˆ y ) =
[ L 1 ( y 1 , ˆ y 1 ) , . . .
, L T ( y T , ˆ y T )] (cid:62) , where L i corresponds to the objective function for 129 the i th task, e.g., cross-entropy loss in case of classiﬁcation.
Constructing an optimal solution for alltasks is often unattainable due to competing objectives.
Hence, an alternative notion of optimalityis used, as described in Deﬁnition 1.
Deﬁnition 1 (Pareto Optimality) .
A point x dominates a point y if L t ( x ) ≤ L t ( y ) for all taskst ∈ [ T ] and L ( x ) (cid:54) =
L ( y ) .
Then, a point x is called Pareto optimal if there exists no point y thatdominates it.
The set of Pareto optimal points forms the Pareto front P L . B A C θθθw 1 =   100 
 w 2 =   010   w 3 =   001   α =   0 . 2   The vector loss function is scalarized by the vector w ∈ [0 , 1] T to form the overall objective w (cid:62) L .
Without loss of generality, we assume that w lies in the T -dimensional simplex ∆ T by imposing the constraint (cid:107) w (cid:107) =
(cid:80)
Tt =1 w t = 1 .
This formulation permits to think of the vector of weights as an encoding of task preferences, e.g., for two tasks letting w =
[0 . 8 , 0 . 2] results in attaching more importance to the ﬁrst task.
Overall, the Multi-Task Learning problem can be formulated within the Empirical Risk Minimization (ERM) framework for preference vector w and dataset D
= { ( x , y ) } i =1 as:
[Equation]
Our overall goal is to discover a low-dimensional parameterization in weight space that yields a(continuous) Pareto Front in functional space.
This desideratum leads us to the following deﬁnition:
Deﬁnition 2 (Pareto Subspace) .
Let T be the number of tasks, X the input space, Y the multitask output space, R ⊂ R N the parameter space, f : X × R
→ Y the function imple-
139 mented by a neural network, and L :
Y × Y → R T> 0 be the vector loss.
Let { θ t ∈ R : 140 t ∈
[ T ] } be a collection of network parameters and S the corresponding convex envelope, i.e.,
S =
(cid:110)(cid:80)
Tt =1 α t θ t : (cid:80)
Tt =1 α t =
and α t ≥ 0 , ∀ t (cid:111) .
Consider the dataset D
=
( D X , D Y ) .
Then,the subspace S is called Pareto if its mapping to functional space via the network architecture fforms a Pareto Front P = L ( f ( D X ; S ) , D Y ) =
{ l : l = L ( f ( D X ; θ ) , D Y ) , ∀ θ ∈ S} .
M ETHOD
We seek to ﬁnd a collection of m neural network models, of identical architecture, whose linearcombination in weight space forms a continuous Pareto Front in objective space .
Model i corresponds to a tuple of network parameters θ i and task weighting w i and implements the functionf ( · ; θ i ) .
We impose connectivity among models by modeling the subspace in the convex hull of theensemble members.
Section 4.1 presents the core of the algorithm, and in Section 4.2 we discussvarious improvements that address Multi-Task Learning challenges.
4. P ARETO M ANIFOLD L EARNING
Let Θ =
[ θ 1 , θ 2 , . .
, θ m ] (cid:62) be an m × N matrix storing the parameters of all models, W = [ w 1 , . . .
, w m ] (cid:62) be a m
×
T matrix storing the task weighting of ensemble members.
By designing the subspace as a simplex, the objective now becomes:
[Equation]
where P is the sampling distribution placed upon the simplex.
In the case where the ensemble members are single-task predictors ( w is one-hot) and the number of tasks coincides with the number of ensemble members ( m = T ) , the matrix of task weightings Wis an identity matrix and Equation 2 simpliﬁes to E ( x , y )
∼D
[Equation] [Equation]
By using the same weighting for both
{+Input : matrix of model parameters Θ = (cid:2) θθ· · · θ T (cid:3) (cid:62) , vector loss function L , train set D , network f , distribution parameters p , window W ∈ N , regularization coefﬁcient ( λ > 0) 
+}
Initialize θ v independently 2 for , y ) ⊆ D do V ← ∅ for i ∈ { 1 , 2 , . . .
, W } do sample α i ∼ Dir ( p ) V ← V ∪ α i θ i ← α (cid:62) i Θ // construct network in convex hull of ensemble members L ( α i ) = (cid:2) L 1 ( α i ) · · · L T ( α i ) (cid:3) ← criterion ( f ( x ; θ i ) , y ) //
compute losses end construct
G t t ∈ [ T ] // see Section 4. [Reviewer qexX]: changed line 11 link to section R ← (cid:80) Tt =1 log (cid:16) 1 |E t | (cid:80) ( α i , α j ) ∈E t exp [ L t ( α i ) − L t ( α j )]
+ (cid:17) // multiforward regularization
(cid:80) Wi = (cid:62) L ( i ) +
· R Backpropagate L total 15 end the losses and the ensemble interpolation, we explicitly associate models and task losses with a oneto-one correspondence, infusing preference towards one task rather than the other and guiding thelearning trajectory to a subspace that encodes such tradeoffs.
Algorithm 1 presents the full training procedure for this ensemble of neural networks, containingmodiﬁcations discussed in subsequent sections.
Figure 1 showcases the algorithm in a toy example [Reviewer qexX]: addressing weakness 2 → mentioned Fig.
1 in the main text.
Concretely, at each training step a random α is sampled and the corresponding convex combinationof the networks is constructed. This procedure is shown in Figure 2.
[Reviewer qexX]: addressing weakness 2 → mentioned Fig.
2 in the main text. The batch is forwardedthrough the constructed network and the vector loss is scalarized by α as well.
The procedure isrepeated W times at each batch (see Section 4. [Table] from Figure 4.2 to Section 4. ) and a regularization term penalizing non-Paretostationary points is added (line 11).
Claim 3.
Let { θ ∗ t ∈ R : t ∈ [ T ] } be the optimal ensemble parameters retrieved at the end of 168 training by Algorithm 1 and let
S be the their convex hull.
Then S is a Pareto Subspace.
Note that we have chosen a convex hull parameterization of the weight space, but there are otheroptions, such as Bezier curves or other nonlinear paths (Wortsman et al., 2021; Draxler et al., 2018).
However, the universal approximation theorem implies no loss of generality for our design choice.
In practice, Claim 3 is validated by uniformly sampling the discovered subspace and the deﬁnitionof a Pareto Subspace is relaxed to conform to the nonconvex settings of Deep Learning, i.e., pointsare called Pareto optimal if the characterization holds in an open neighborhood rather than globally. [Reviewer [Table] Addressing
[Table]
mality and [Table] [Table] 4. R EGULARIZATION AND BALANCING
Loss and gradient balancing schemes.
A common challenge in Multi-Task Learning is the casewhere tasks have different loss scales, e.g., consider datasets with regression and classiﬁcation taskssuch as UTKFace .
Then, using the same weighting α for both the losses and the weight ensembling, as presented in Equation 2, the easiest tasks are favored and the important property of scaleinvariance is neglected.
To prevent this, the loss weighting needs to be adjusted.
Hence, we propose simple balancing schemes: one loss and one gradient balancing scheme, whose effect is towarp the space of loss weightings.
While gradient balancing schemes are applied on the sharedparameters, loss balancing also affects the task-speciﬁc decoders, rendering the methodologies canbe complementary.
To avoid cluttering, balancing schemes are not presented in Algorithm 1.
In terms of loss balancing, we use a lightweight scheme of adding a normalization coefﬁcient to each loss term which depends on past values.
Concretely, let W ∈ Z + be apositive integer and L m ( τ 0 ) be the loss of task m in step τ 0 .
Then, the regularization coefﬁcient is L ( τ [Equation]
; W [Equation] L m ( τ 0 + 1 − τ ) for τ 0 ≥ W resulting in the overall loss
[Equation]
L m ( τ 0 ; W ) .
For gradient balancing.
let g t be the gradient of task
[Figure] t ∈ [ T ] w.r.t.
the shared parameters.
Previously, the update rule occurred with the overall gradientg total =
α (cid:62) G = α (cid:62) [ g. . .
g T ] .
We impose a unit (cid:96) 2 -norm for gradients and perform theupdate with (cid:101) g
[Equation]
[Equation] [Equation]
g T ] where
[Equation]
Improving stability by Multi-Forward batch regularization.
Consider two different weightingsα 1 and α 2 ∈ ∆ T − 1 .
Without loss of generality [ α 1 ] 0 = α 1 > [ α 2 ] 0 = α 2 .
Then, ideally, the 195 interpolated model closer to the ensemble member for task 1 has the lowest loss on that task, i.e., wewould want the ordering L 1 ( α 1 ) <
L 1 ( α 2 ) , and, equivalently for the other tasks.
Furthermore, ifα = [1 − (cid:15), (cid:15) / T − 1 , . . .
, (cid:15) / T − 1 ] , only one member essentially reaps the beneﬁts of the gradient updateand moves the ensemble towards weight conﬁgurations more suitable for one task but, perhaps deleterious for the remaining ones.
Thus, we propose repeating the forward pass W times for differentrandom weightings { α i } i ∈
[ W ] , allowing the advancement of all ensemble members concurrently in 201 a coordinated way.
By performing multiple forward passes for various weightings, we achieve alower discrepancy sequence and reduce the variance of such pernicious updates.
We also include a regularization term, which penalizes the wrong orderings and encourages the subspace to have Pareto properties.
Let V be the set of interpolation weighs sampled in the current batch V = { α w = ( α w, 1 , α w, 2 , . . .
, α w,T ) ∈
∆ T − 1 } w ∈ [ W ] .
Then each task deﬁnes the directed graph G t =
( V , E t ) where E t = { ( α i ,
α j ) ∈
V × V : α i,t < α j,t } .
The overall loss becomes:
[Equation]
The current formulation of the edge set penalizes heavily the connections from vertices with lowvalues.
For this reason, we only keep one outgoing edge per node, deﬁned by the task lexicographicorder, resulting in the graph G LEX t =
( V , E LEX t ) and |E LEX t | =
W − 1 , ∀ t ∈ [ T ] .
Note that the regu- 206 larization term is convex as the sum of log-sum-exp terms.
If no violations occur, the regularizationterm is zero.
Figure 3 offers a visual explanation of the regularization term.
The role of sampling.
Another component of Algorithm 1 is the sampling imposed on the convexhull parameterization.
During training, the sampling distribution dictates the loss weighting usedand, hence, modulates the degree of task learning.
A natural choice is the Dirichlet distribution
Dir ( p ) where p ∈ R T> 0 are the concentration parameters, since its support is the T -dimensionalsimplex ∆ T .
For p = p 1 T , the distribution is symmetric; for p <
1 the sampling is moreconcentrated near the ensemble members, for p > 1 it is near the centre and for p {+= 
+}
it correspondsto the uniform distribution.
In contrast, for p 1 (cid:54) =
p 2 the distribution is skewed.
In our experiments,we use symmetric Dirichlet distributions with p ≥ 1 to guide the ensemble to representations bestsuited for Multi-Task Learning.
[Figure]
We have used a different color for each seed of PML.
Baselines are shown in shades of gray: scatter plot for MTL baselines and dashed lines for single task.
In both datasets, Pareto Manifold Learning discovers subspaces with diverse and Pareto-optimal solutions and outperforms the baselines.
E XPERIMENTS
We evaluate our method on several datasets, such as MultiMNIST , Census , MultiMNIST-3 ,
UTKFace and CityScapes , and various architectures, ranging from MultiLayer Perceptrons(MLPs) to Convolutional Neural Networks (CNNs) and Residual Networks (ResNets).
Eachensemble member is initialized independently.
In all experiments, the learning rate for our methodis m -fold the learning rate of the baselines to counteract the fact that the backpropagation stepscales the gradients by m − 1 in expectation.
The detailed settings used for each dataset andadditional experiments are provided in the appendix.
Our overarching objective is to constructcontinuous weight subspaces which map to Pareto Fronts in the functional space.
However, ourmethod produces a continuum of results rather than a single point, rendering tabular presentationcumbersome.
For this reason, (a) for tables we present the best-of-(sampled)-subspace results, (b)we experiment on numerous two-task datasets where plots convey the results succinctly, (c) presentqualitative results on three-task datasets.
The source code will be released after the review process.
Baselines.
We explore various algorithms from the literature: 1.
Single-Task Learning (STL),
Linear Scalarization LS) which minimizes the average loss 1 T (cid:80) {+Tt =
+} L t , 3.
Uncertainty Weighting (UW, Cipolla et al. 2018) Multiple
-gradient descent algorithm (MGDA, Sener & Koltun 2018)
Dynamic Weight Averaging (DWA, Liu et al. 2019)
Projecting Conﬂicting Gradients (PCGrad, Yu et al. 2020)
Impartial Multi Task Learning (IMTL, Liu et al. 2020),
Conﬂict-Averse Gradient Descent CAGrad , Liu et al. 2021) and 9.
Bargaining Multi-Task Learning (Nash-MTL, Navon et al. 2022).
5. E XPERIMENTS ON D ATASETS WITH TWO CLASSIFICATION TASKS
In this section, we focus on datasets with two tasks, both classiﬁcation.
This setting allows for richvisualizations that we use to draw insights on the inner workings of the algorithms.
MultiMNIST .
We investigate the effectiveness of Pareto Manifold Learning on digit classiﬁcationusing a LeNet model with a shared-bottom architecture.
The ensemble consists of two memberswith single task weightings.
To gauge the performance of the models lying in the linear segmentbetween the nodes, we test the performance on the validation set on the ensemble members as wellas for 9 models uniformly distributed across the edge, resulting in 11 models in total.
We use thisevaluation/plotting scheme throughout the experiments.
We ablate the effect of multi-forward training on Appendix D; we use a grid search on window W ∈ { 2 , 3 , 4 , 5 } and strength λ ∈ { 0 , 2 , 5 , 10 } 247 along with the base case of ( W, λ ) =
(1 , 0) and present in the main text the setting that achieves
the [Figure]
[Figure]
[Figure] highest mean (across seeds)
HyperVolume score on the validation set.
Figure 4 shows the results on
MultiMNIST using multi-forward regularization with window W = 4 and strength λ = 0 .
Weobserve that most baselines are characterized by limited functional diversity; their predeﬁned optimization schemes lead the differently seeded/initialized training runs to ﬁnal models with similarperformance (same markers are clustered in the plots). This lack of functional diversity, as well asinability to consistently outperform the Linear Scalarization baseline, are also noted by Kurin et al.(2022); Xin et al.
[Table] In contrast, all Pareto Manifold Learning seeds ﬁnd subspaces with diverse 255 functional solutions.
This statement is quantitatively translated to higher HyperVolume comparedto the baselines, shown in Table 4 of the appendix, and can be attributed to the observation that
Equation 2 generalizes the Linear Scalarization method.
Census .
We explore the method on the tabular dataset Census (Kohavi, 1996) using a MultiLayer Perceptron.
We focus on the task combination of predicting age and education level, similarto Ma et al.
We perform the same ablation study as before and present the results on Figure 4for the best setting ( W = 3 and λ = 10 ).
In the case of MultiMNIST , there exists symmetrybetween the tasks, both digits are drawn from the same distribution and placed in the pixel grid ina symmetric way, resulting in equal pace learning.
However, in the case of Census , tasks differ instatistics and, yet, the proposed method recovers a Pareto subspace with diverse solutions.
5. B EYOND P AIRS OF C LASSIFICATION T ASKS : M ULTI MNIST-3 AND UTKF ACE
We expand the experimental validation to triplets of tasks, consider regression and more complex architectures, graduating from MLPs and CNNs to ResNets (He et al., 2016).
For three tasks, we createa 2D grid of equidistant points spanning the three single-task predictors.
If n is the number of interpolated points between two (out of three) members, the grid has (cid:0) n + (cid:1) points.
We use n = 11 , resulting in 66 points.
For visual purposes, neighboring points are connected.
For three tasks, it would bevisually cluttering to present the discovered subspaces with multiple seeds and baselines.
Hence, weopt for a more qualitative discussion in this section and present quantitative ﬁndings in the appendix.
MultiMNIST-3 .
First, we construct an equivalent of MultiMNIST for 3 tasks.
Digits are placedon top-left, top-right and bottom-centre.
Figure 5a shows the results on MultiMNIST-3 .
Asargued previously, MNIST variants are characterized by task symmetry and Figure 5a reﬂects this.
For this reason, we do not employ any balancing scheme.
The 3D plot in conjunction with the
[Table]
simplices reveal that the method has the effect of gradual transfer of learned representation from onemember to the other, and offers a succinct visual conﬁrmation of Claim 3.
UTKFace .
The UTKFace dataset (Zhang et al., 2017) has more than 20,000 face images andthree tasks: predicting age (modeled as regression using Huber loss - similar to (Ma et al., 2020)),classifying gender and ethnicity.
The introduction of a regression task implies that losses have vastlydifferent scales, which dictates the use of balancing schemes, as discussed in Section 4.2.
We applythe proposed gradient-balancing scheme and present the results in Figure 5b.
For visual unity and toremain in the theme of “higher is better”, the negative Huber loss is plotted.
Despite the increasedcomplexity and the existence of a regression task, the proposed method discovers a Pareto Subspace .
Additional experiments and qualitative results are provided in Appendix G.
5. S CENE UNDERSTANDING
We also explore the applicability of Pareto Manifold Learning for CityScapes (Cordts et al.,2016), a scene understanding dataset containing high-resolution images of urban street scenes.
Ourexperimental conﬁguration is drawn from Liu et al.
; Yu et al.
; Liu et al.
(2021); Navonet al.
(2022) with some modiﬁcations.
Concretely, we address two tasks: semantic segmentation anddepth regression.
We use a SegNet architecture (Badrinarayanan et al., 2017) trained for 100 epochswith Adam optimizer (Kingma & Ba, 2015) of initial learning rate 10 − 4 , which is halved after 75epochs.
The images are resized to 128 × 256 pixels.
In the initial training steps any samplingα results in a random model, due to initialization, and the algorithm has a warmup period untilthe ensemble members have acquired meaningful representations.
Hence, to reduce computationaloverhead and help convergence, the concentration parameter of the Dirichlet distribution is set [-to p 
-]top 0 = 5 .
We use gradient balancing, window W = 3 and λ = 1 .
The results are presented in Table 1.
In Depth Estimation and out of MTL methods, Pareto Manifold Learning is near-optimal with
MGDA narrowly better.
However, the performance compared to the other algorithms is superior.
In Semantic Segmentation, our method outperforms MGDA, but is worse than other baselines.
Overallno multi-task method dominates Pareto Manifold Learning. [Table] It is remarkable that, despite our goal ofdiscovering Pareto subspaces , the proposed method is on par in performance on Semantic Segmentation with the state-of-the-art algorithms, and better than the vast majority on Depth Estimation.
C ONCLUSION
In this paper, we proposed a weight-ensembling method tailored to Multi-Task Learning; multiplesingle-task predictors are trained in conjunction to produce a subspace formed by their convex hull,and endowed with desirable Pareto properties.
We experimentally show on a diverse suite of benchmarks that the the proposed method is successful in discovering Pareto subspaces and outperformssome state-of-the-art MTL methods.
An interesting future direction is to perform a hierarchicalweight ensembling, sharing progressively more of the lower layers, given that the features learnedat low depth are similar across tasks.
An alternative exploration venue is to connect our method tothe challenge of task afﬁnity (Fifty et al., 2021; Standley et al., 2020) via a geometrical lens of theloss landscape.
A A PPENDIX O VERVIEW
As a reference, we provide the following table of contents solely for the appendix.
A. Appendix Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Appendix A
B . Details on experimental conﬁgurations
Appendix B C . Details on Illustrative Example
Appendix C
D.
on Multi Forward Regularization
Appendix D
E
HyperVolume Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.Appendix E
F
. Additional experiments on MultiMNIST-
Appendix F
G
. Additional
experiments on UTKFace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Appendix
G
H
Details
sampling. . .
.Appendix H I
NEW! Connection between
Appendix J
K. NEW! Additional experiments
. Appendix K
B E XPERIMENTAL D ETAILS
MultiMNIST MultiMNIST is a synthetic dataset derived form the samples of MNIST .
Sincethere is no publicly available version, we create our own by the following procedure.
For each
MultiMNIST image, we sample (with replacement) two MNIST images (of size 28 × 28 ) and placethem top-left and bottom-right on a 36 × 36 grid.
This grid is then resized to 28 × 28 pixels.
Theprocedure is repeated 60000 times, 10000 and 10000 times for training, validation and test datasets.
We use a LeNet shared-bottom architecture.
Speciﬁcally, the encoder has two convolutional layerswith 10 and 20 channels and kernel size of 5 followed by Maxpool and a ReLU nonlinearity each.
The ﬁnal layer of the encoder is fully connected producing an embedding with 50 features.
Thedecoders are fully connected with two layers, one with 50 features and the output layer has 10.
Weuse Adam optimizer Kingma & Ba (2015) with learning rate 10 − 3 , no scheduler and the batch sizeis set to 256.
Training lasts 10 epochs.
Census
The original version of the Census (Kohavi, 1996) dataset has one task: predictingwhether a person’s income exceeds $50000.
The dataset becomes suitable for Multi-Task Learningby turning one or several features to tasks (Lin et al., 2019).
We focus on the task combination ofpredicting age and education level, similar to Ma et al.
The model has a Multi-Layer Perceptron shared-bottom architecture.
The encoder has one layer with 256 neurons, followed by a ReLUnonlinearity, and two decoders with 2 output neurons each (since the tasks are binary classiﬁcation).
Training lasts 10 epochs.
We use Adam optimizer learning rate of 10 − 3 .
MultiMNIST-
The conﬁguration of MultiMNIST is used.
Now, the model has three decodersand training lasts 20 epochs.
UTKFace
The UTKFace dataset has more than 20,000 face images of dimensions 200 × 200pixels and 3 color channels.
The dataset has three tasks: predicting age (modeled as regression using
Huber loss - similar to (Ma et al., 2020)), classifying gender and ethnicity (modeled as classiﬁcationtasks using Cross-Entropy loss).
Images are resized to 64 × 64 pixels, age is normalized and a 80/20train/test split is used.
We use a shared-bottom architecture; the encoder is a ResNet18 (He et al.,2016) model without the last fully connected layer.
The decoders (task-speciﬁc layers) consist of onefully-connected layer, where the output dimensions are 1, 2 and 5 for age (modeled as regression),gender (binary classiﬁcation) and ethnicity (classiﬁcation with 5 classes).
Training lasts 100 epochs,batch size is
and we use Adam optimizer with a learning rate of 10 − 3 .
No scheduler is used.
CityScapes Our experimental conﬁguration is very similar to prior work, namely Liu et al.(2019); Yu et al.
; Liu et al.
; Navon et al.
All images are resized to 128 × 256 .
The tasks used are coarse semantic segmentation and depth regression.
The task of semantic segmentation has 7 classes, whereas the original has 19.
We use a SegNet architecture (Badrinarayananet al., 2017) and train the model for 100 epochs with Adam optimizer (Kingma & Ba, 2015) of aninitial learning rate 10 − 4 .
We employ a scheduler that halves the learning rate after 75 epochs.
C D ETAILS OF THE ILLUSTRATIVE EXAMPLE
The details of the illustrative example are provided in this section.
We use the conﬁguration presented by Navon et al.
(2022), which was introduced with slight modiﬁcations by Liu et al.
(2021)and Yu et al.
Speciﬁcally, let θ =
( θ 1 , θ
) ∈ R 2 be the parameter vector and L =
(˜ (cid:96) 1 , ˜ (cid:96) 2 )be the vector objective deﬁned as follows:
[Equation]
and
[Equation]
where
[Equation]
and
[Equation]
We use the experimental setting outlined by (Navon et al., 2022) with minor modiﬁcations, i.e., Adam optimizer with a learning rate of 2 e − 3 and training lasts for 50 K iterations.
Theoverall objectives are (cid:96) 1 = c · ˜ (cid:96) 1 and (cid:96) 2 = ˜ (cid:96) 2 where we explore two conﬁgurations for thescalar c , namely c ∈ { 0 . 1 , 1 } . For c = 1 , the two tasks have losses at the same scale. For c = 0 .
1 , the difference in loss scales makes the problem more challenging and the algo- rithm used should be characterized by scale invariance in order to ﬁnd diverse solutions spanning the entirety of the Pareto Front.
The initialization points are drawn from the following set{ ( − 8 .
, (9 . 0 , 9 . 0) , ( − 7 . 5 , − 0 . 5) , (9 , − 1 . 0) } .
In the case of Pareto Manifold Learn- 524 ing with two ensemble members there are 5 2 = 25 initialization pairs.
In the main text we use theinitialization pair with the worst initial objective values.
Figure 6 presents the results for the case of different loss scales, i.e., c = 0 .
We plot variousbaselines and three versions of the proposed algorithm, Pareto Manifold Learning or PML in short. We focus on the effect of the balancing schemes, introduced in Section 4.2, resulting in the use of nobalancing scheme (denoted as PML), the use of gradient balancing (denoted as PML-gb) and the useof loss balancing (denoted as PML-lb).
We dedicate two ﬁgures for each version of the algorithmand we present all 25 initialization pairs for completeness.
Figure 7 and Figure 8 correspond tono balancing scheme in the case of equal loss scales c = 1 . 0 , i.e., they complement Figure 1 ofthe main text. The subsequent ﬁgures focus on the case of unequal loss scales where c = 0 .
Figure 9 Figure 10 correspond to no balancing scheme, Figure 11 and Figure 12 correspond to theuse of gradient balancing, Figure 13 and Figure 14 correspond to the use of loss balancing.
The ﬁrstﬁgures of each pair show the trajectories for each initialization pair, with markers for initial and ﬁnalpositions.
The other ﬁgures of each pair dispense of the visual clutter and focus on the subspacediscovered in the ﬁnal step of training, which is plotted with dashed lines along with the analytical Pareto Front in solid light blue.
Hence, they provide a succinct overview of whether the method wasable or not to discover the (entire) Pareto Front.
For c = 1 . 0 , the proposed method is able to retrieve the exact Pareto Front with no balancing schemefor most initialization pairs, as can be seen in Figure 8. In three cases (out of 25), the method fails.
In our experiments, we found that allowing longer training times or higher learning rates resolvethe remaining cases.s
For c = 0 . 1 , the problem is more challenging and the vanilla version of thealgorithm results in a subset of the analytical Pareto Front.
Figure 10 shows that this subset isconsistent across initialization pairs, excluding the ones the method fails, and focuses on the taskwith higher loss magnitude.
Applying gradient balancing, shown in Figure 11 and Figure 12, allowsthe method to retrieve (a superset of) the Pareto Front for all initialization pairs.
Similarly, lossbalancing, shown in Figure 13 and Figure 14, results in the exact Pareto Front.
Hence, the inclusionof balancing schemes endows scale invariance in the proposed algorithm.
Balancing schemes areused for the more challenging datasets, such as CityScapes . Applying balancing schemes resolve these issues.
α [Figure]
[Figure]
α 1 αα
[Figure]
ααα
[Figure] [Figure]
[Figure] [Figure]
[Figure]
[Figure]
D A BLATION ON M ULTI -F ORWARD R EGULARIZATION
Multi-Forward regularization, introduced in Section 4.2, penalizes the ensemble if the interpolatedmodels’ losses (sampled within a batch) are not in accordance with the tradeoff imposed by thecorresponding interpolation weights.
Simply put, the closer we sample to the member correspondingto task 1, the lower the loss should be on task 1.
The same applies to the other tasks. Equation 3in the main text presents the case of two tasks, where the idea of the regularization is outlined inloss space. For completeness, we present the underlying graph construction for the cases of two andthree tasks in Figure 15 and Figure 16, respectively. The nodes of the graphs are associated withthe sampled weightings and the edges for the graph G t of task t are drawn w.r.t.
the correspondingpartial ordering. If the loss ordering is violated for a given edge, a penalty term is added. We ablate the effect multi-forward training and the corresponding regularization have on performance.
We explore the MultiMNIST and Census datasets using the same experimental conﬁgurations as in the main text.
We are interested in two parameters:
• W : number of α re-samplings per batch.
This parameter is also referred as window .
• λ : the regularization strength as presented in Algorithm 1. For λ = no regularization is
applied but the subspace is still sampled W times and the total loss takes into account all the respective interpolated models.
Figure 17 and Table 2 present the results for MultiMNIST .
Figure 18 and Table 3 present the results for Census .
It is important to note that MultiMNIST is symmetric, while Census is not.
As a result, the features learned for each single-task predictor are helpful to one another and the caseof λ = 0 , i.e., no regularization and only multi-forward training, is beneﬁcial for MultiMNIST butnot for Census . Intuitively, both digit classiﬁcation tasks have the same difﬁculty and posteriordistribution, which produces few violations of monotonicity constraints and renders the regularization less applicable.
[Reviewer qexX]: expanded commentary on λ = 0 for
MultiMNIST .
On the other hand, severe regularization such as λ = 10 can be harmful andhinder training.
More details in table and ﬁgure captions.
[Table]
[Table]
[Table]
W = 3 λ =0.
λ
=0.
λ
=0.
λ
=0.
W = 4 λ =0.
λ
=0.
λ
=0.
λ
=0.
W = 5 λ =0.
λ
=0.
λ
=0.
λ
=0.
[Table]
-MTL (m) W = 4 and λ =0 .0 .0 .0 .0 .0 .
97 top-leftaccuracy 0 .0 .0 .0 .0 .0 .b
o tt o m - r i g h t a cc u r a c y
SingleTaskPML-0PML-1PML-2LSUW
MGDADWAPCGradIMTLCAGradNash-MTL (n) W = 5 and λ =0 .0 .0 .0 .0 .0 .
97 top-leftaccuracy 0 .0 .0 .0 .0 .0 .b
o tt o m - r i g h t a cc u r a c y
SingleTaskPML-
[Figure]
PML
[Figure]
PML-
[Figure]
LSUW
MGDADWAPCGradIMTLCAGradNash-MTL (o) W = 5 and λ =0 .0 .0 .0 .0 .0 .
97 top-leftaccuracy 0 .0 .0 .0 .0 .0 .b
o tt o m - r i g h t a cc u r a c y
SingleTaskPML-0PML-1PML-2LSUW
MGDADWAPCGradIMTLCAGradNash-MTL (p) W = 5 and λ =0 .0 .0 .0 .0 .0 .
[Figure]
leftaccuracy 0 .0 .0 .0 .0 .0 .b
o tt o m - r i g h t a cc u r a c y
[Figure]
0PML-
[Figure]
PML
[Figure]
2LSUW
MGDADWAPCGradIMTLCAGradNash-MTL (q) W = 5 and λ = regularization coefﬁcient λ on the validation dataset.
The case of no multi-forward ( W = 1 ) is presented in the ﬁrst row.
Multi-forward regularization for higher W values is beneﬁcial.
Intuitively, attaching serious weight on the regularization λ ∈ { 5 , 10 } while sampling few times W ∈ { 2 , 3 } leads to suboptimal per-
formance since the update step focuses on an uninformed regularization term.
The accompanying quantitative analysis appears in Table 2.
[Table]
.0 .e
du c a t i o n a cc u r a c y
SingleTaskPML(ours)-0PML(ours)-1PML(ours)-2LSUW
MGDADWAPCGradIMTLCAGradNash-MTL (o) W = 5 and λ =0 .
830 0 . 832 ageaccuracy 0 .0 .0 .0 .0 .0 .0 .
.e
du c a t i o n a cc u r a c y
SingleTaskPML(ours)-0PML(ours)-1PML(ours)-2LSUW
MGDADWAPCGradIMTLCAGradNash-MTL (p) W = 5 and λ =
[-830 ageaccuracy 0 .0 .0 .0 .0 .0 .
-]
[-du c a t i o n a cc u r a c y SingleTaskPML-0PML-1PML-2LSUW MGDADWAPCGradIMTLCAGradNash-MTL (o) W = 5 and λ =0 . 830 ageaccuracy 0 .0 .0 .0 .0 .0 .
-][Figure]
[Figure]
[Figure]
[Figure]
optimal point reference point 0 0 .
E H YPER V OLUME ANALYSIS ON M ULTI MNIST AND C ENSUS
HyperVolume is a metric widely used in multi-objective optimization that captures the quality ofexploration.
A visual explanation of the metric is given in Figure 19.
Table 4 presents the resultsof Figure 4 of the main text in a tabular form.
We present the best three results per column (higheris better) to succinctly and visually show that all Pareto Manifold Learning seeds outperform thebaselines.
MultiMNIST
Census
Task
[Table]
F M ULTI MNIST-3 ADDITIONAL RESULTS
This section serves as supplementary to Section 5.2 of the main text.
MultiMNIST-3 isa synthetic dataset generated by MNIST samples in a manner similar to the creation of the
MultiMNIST dataset, which is ubiquitous in the Multi-Task Learning literature.
Speciﬁcally, each
MultiMNIST-3 sample is created with the following procedure.
Three randomly sampled digitsof size 28 × 28 are placed in the top-left, top-right and bottom middle pixels of a 42 × 42 grid.
For the pixels where the initial digits overlap, the maximum value is selected. Finally, the image isresized to 28 × 28 pixels. Figure 20 shows some examples of the dataset, which consists of threedigit classiﬁcation tasks.
Section 5 compares the performance of baselines and the proposed method while Figure 21 presentsvisually the performance achieved on the discovered subspace.
[Table]
[Figure]
G UTKF ACE ADDITIONAL RESULTS
This section serves as supplementary to Section 5.2.
Section 6 compares the performance of thebaselines and the proposed method.
We experiment without balancing schemes and with gradientbalancing, and present the results in Figure 22 and Figure 23, respectively.
Together with the quantitative results, we observe that for datasets with varying task difﬁculties, scales, etc. the lack ofbalancing can be impeding.
On the other hand, its inclusion makes the subspace functionally diverse and boosts overall performance.
For instance, Huber loss on the task of age prediction issigniﬁcantly improved.
[Table]
[Figure]
H ub e r l
o
s s – A g e
[Figure]
A c c u r a c y –
G e nd e r
[Figure]
A cc u r a c y – R a ce
[Figure]
[Figure] [Figure]
G
[Figure]
nd
[Figure]
cc
[Figure]
r a
[Figure]
[Figure]
[Figure]
H D ETAILS ON SAMPLING
[Figure]
This appendix expands on Section 4.2 and, speciﬁcally, presents in greater detail the intuition behind [Figure] the sampling distribution’s parameters. Let p ∈ R T + be the parameters of the Dirichlet distribution.
[Figure]
Assuming no prior knowledge on the tasks, e.g., task difﬁculties or afﬁnities, a symmetric distribu-
[Figure]
tion is used by setting p = p 1 T .
This design choice results in three cases:
[Figure]
• p = 1 : the distribution is uniform on the simplex.
Intuitively this means that all tasks are
[Figure]
equally important
and we care about the diversity of solutions for all tradeoffs (reﬂected in
the linear scalarization weights)
• p ∈ (0 , 1) : the distribution is more concentrated towards the ensemble members, as in thetop row of Figure 24.
Assume an extreme case of two tasks and p0.
Then the distribution
degenerates to a Bernoulli distribution.
Effectively, at each iteration one of the ensemble
members is selected and its weights are updated, which will result in two separate and inde-
pendent single-task predictors with no common representation infused about the other task.
Then, linearly interpolating in weight space will result in models with random predictions
[Table]
for both tasks, since the training procedure has not focused in retrieving a Pareto Subspace.
[Table]
For milder cases (e.g. p = 0 . 7 ) , we observed that the models in the middle of the linear
[Table]
interpolation suffered in performance which can be attributed to the fact that the samplingfocused more on single-task rather than multi-task representations and performance.
[Table]
• p > 1 .
Then the distribution is more concentrated towards the midpoint of the simplex, as inthe bottom row of Figure 24.
Assume an extreme case of two tasks and p → ∞ .
Then, thedistribution becomes deterministic and outputs equal weights for all tasks.
The randomly
[Table]
and independently initialized ensemble members will collapse to each other, resulting induplicate ensemble members.
Similarly, for very large values (e.g. p = 100 ), the functionaldiversity of the ensemble will suffer since the weights produced by the distribution will be
[Table]
almost equal for all tasks, resulting in a milder version of the aforementioned phenomenon.
[Table]
In contrast, we found that small values such as p = 2 or p = 3 can help convergence since
[Table] [Figure] i [Figure]
a cc u r a c y
[Figure]
[Figure]
[Figure]
[Figure]
a t i o n
[Figure]
cc u r a c y
[Figure]
[Figure]
they put more emphasis towards common representation (compared to p = 1 ), but maylimit functional diversity.
Figure 25 presents experimental results on MultiMNIST and Census for various concentrationparameters p ∈ { 0 , 0 . 1 , 100 } of the Dirichlet distribution.
{+Let θ 
+}
and θ 2 be the parameters of 633 the ensemble members.
For p = 0 , the ensemble consists of two single-task predictors with nomultitask learning representational knowledge, since their interpolation meets a low accuracy/highloss barrier.
We omit the case of p = 0 for Census for visual clarity.
This lack of commonrepresentation is evident in the cosine similarities as well, where for p = 0 cos( θ 1 , θ 2 ) ≈ 0 .
On theother hand, for p = 0 . 1 , common representations are infused into the ensemble and the experimentalresults show that the test performance is characterized by diversity.
However, this comes at theexpense of the interpolated models at the middle of the line segment, where the performance issuboptimal compared to p = 100 for MultiMNIST .
This behavior is also illustrated in the cosinesimilarities, where for p = 100 the ensemble weights α are in an (cid:15) -ball around the midpoint causingthe independently initialized models to progressively collapse.
For Census , we also observe thatthis collapsing leads to very high cosine similarity cos( θ 1 , θ 2 ) > 0 .
9 and the ensemble is suboptimalcompared to p = 0 .
I C ONNECTION BETWEEN P ARETO O PTIMALITY AND MULTIPLE VALLEYINTERSECTIONS
[Reviewer qexX]: Added ap- pendix regarding weakness 1.
For clarity during the rebuttal, this discussion has been added as a standalone appendix.
It will be incorportared in the appendix regarding the illustra- tive example.
In this section, we investigate the connection between the intersection of multiple loss landscapes,pareto optimality and the effect of the proposed algorithm Pareto Manifold Learning.
We use theillustrative example, presented in Figure 1.
Let Θ be the parameter space of the model and L t :Θ → R , t ∈ { 1 , 2 } , be the losses of the problem.
For α ∈ [0 , 1] and θ ∈ Θ , the overall objective is
L ( θ , α ) =
α L 1 ( θ ) + (1 − α )
L 2 ( θ ) .
Figure 26 and the accompanying Figure 27 present the overall loss objective as α varies from 0 to1.
For the extreme values of the range, the loss landscape is inherently single-task.
The subspacediscovered by the method is depicted in blue, while a black ‘x’ is used for the corresponding interpolated model, i.e., it corresponds to L ( α θ 1 + (1 − α )
θ 2 , α ) .
Figure 28 presents the overall losseson the subspace by ﬁxing as a function of one of the parameters.
In other words, the proposedmethod tracks the optimum in parameter space as the overall objective evolves and the various losslandscapes are weighted accordingly.
While an acceptable multi-task solution lies in the intersectionof low loss landscapes, Pareto Manifold Learning focuses on the aforementioned dynamic scenarioof loss weighting.
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
J A DDITIONAL R ELATED W ORK
[Reviewer
[Table]
In this appendix, we further expand on prior work.
Linear mode connectivity, as in (Wortsman et al.,2021), encourages ﬂatness and, therefore, is linked with methods explicitly enforcing ﬂat minima(Chaudhari et al., 2017;
Foret et al., 2021;
Dinh et al., 2017; Jiang* et al., 2020).
These approachesare applicable when designing a single objective, e.g. average of losses in Multi-Task Learning,but do not allow for the infusion of Pareto properties and the inclusion of tradeoffs.
[Table]
669 (2018) produce ﬂat minima by averaging multiple weight vectors discovered during the optimizationtrajectory, so that the ﬁnal model lies in the middle of the low-loss basin.
Wortsman et al.
(2022)perform weight ensembling with ﬁne-tuned models produced via different hyperparameter conﬁgurations.
Apart from the recent weight ensembling works, output ensembling has been one of thestaples of machine learning literature.
Lakshminarayanan et al.
(2017) utilize deep ensembles foruncertainty prediction but inference scales linearly with the number of ensemble members.
Wenet al.
(2020) improve on the computational complexity of output ensembles by sharing the bulk ofthe parameters among members and differentiating them via rank-1 matrices, while Havasi et al.(2021) employ a multi-input multi-output network by accommodating independent subnetworks foreach ensemble and allowing a single-forward pass ensemble prediction.
However, this results insubnetworks with incompatible architecture which does not allow for a continuous approximationof the Pareto Front.
[Table]
[Figure]
K A DDITIONAL EXPERIMENTS
[All reviewers]: Added ap- pendix with additional baselines.
In this section
[Figure]
we supplement our experimental ﬁndings on MultiMNIST with additional baselines, namely HPN-LN and HPN-EPO (Navon et al., 2021) and COSMOS (Ruchte & Grabocka,2021) 1 .
We use hyperparameters of Ruchte & Grabocka (2021) for both methods.
We provide twoexperimental settings:
• Setting I : 10 epochs and no learning rate scheduler, i
[Figure]
e., the setting used for all other meth-
ods in Figure
• Setting II :the experimental setting used by (Ruchte & Grabocka, 2021),
i
e., 100 epochs forCOSMOS and 150 epochs for HPN-LN/HPN-EPO with multi-step learning scheduler.
Figure 29 presents the results with the additional baselines, using three seeds each.
We use dashedlines for setting I and solid lines for setting II and group the three methods in various color shades(blue, green, red) for visual clarity.
We observe that in the original setting of 10 epochs, all new baselines are suboptimal compared to all methodologies.
For setting II , the hypernetwork methodologiesare competitive with some baselines but are suboptimal compared to the proposed method.
For
COSMOS, only one seed is competitive with the proposed method.
Moreover, HPN-LN, HPN-EPOemploy a hypernetwork of 1.6m parameters, while the target network has < 50 k parameters.