Original: In Defense of the Unitary Scalarization for Deep Multi-Task Learning
Revised: In Defense of the Unitary Scalarization for Deep Multi-Task Learning

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Recent multi-task learning research argues against unitary scalarization , wheretraining simply minimizes the sum of the task losses.
Revised: Recent multi-task learning research argues against unitary scalarization , where training simply minimizes the sum of the task losses.

Original: Several ad-hoc multi-task optimization algorithms have instead been proposed, inspired by various hypothesesabout what makes multi-task settings difﬁcult.
Revised: Several ad-hoc multi-task optimization algorithms have instead been proposed, inspired by various hypotheses about what makes multi-task settings difﬁcult.

Original: The majority of these optimizersrequire per-task gradients, and introduce signiﬁcant memory, runtime, and implementation overhead.
Revised: The majority of these optimizers require per-task gradients, and introduce signiﬁcant memory, runtime, and implementation overhead.

Original: We show that unitary scalarization, coupled with standardregularization and stabilization techniques from single-task learning, matches orimproves upon the performance of complex multi-task optimizers in popular supervised and reinforcement learning settings.
Revised: We show that unitary scalarization, coupled with standard regularization and stabilization techniques from single-task learning, matches or improves upon the performance of complex multi-task optimizers in popular supervised and reinforcement learning settings.

Original: We then present an analysis suggestingthat many specialized multi-task optimizers can be partly interpreted as forms ofregularization, potentially explaining our surprising results.
Revised: We then present an analysis suggesting that many specialized multi-task optimizers can be partly interpreted as forms of regularization, potentially explaining our surprising results.

Original: We believe our resultscall for a critical reevaluation of recent research in the area.
Revised: We believe our results call for a critical reevaluation of recent research in the area.

Original:  Introduction
Revised:  Introduction

Original: Multi-Task Learning (MTL) [5] exploits similarities between tasks to yield models that are moreaccurate, generalize better and require less training data.
Revised: Multi-Task Learning (MTL) [5] exploits similarities between tasks to yield models that are more accurate, generalize better and require less training data.

Original: Owing to the success of MTL on traditionalmachine learning models [3, 16, 22] and of deep single-task learning across a variety of domains, agrowing body of research has focused on deep MTL.
Revised: Owing to the success of MTL on traditional machine learning models [3, 16, 22] and of deep single-task learning across a variety of domains, a growing body of research has focused on deep MTL.

Original: The most straightforward way to train a neuralnetwork for multiple tasks at once is to minimize the sum of per-task losses.
Revised: The most straightforward way to train a neural network for multiple tasks at once is to minimize the sum of per-task losses.

Original: Adopting terminologyfrom multi-objective optimization, we call this approach unitary scalarization .
Revised: Adopting terminology from multi-objective optimization, we call this approach unitary scalarization .

Original: While some work shows that multi-task networks trained via unitary scalarization exhibit superiorperformance to independent per-task models [28, 34], others suggest the opposite [29, 53, 57].
Revised: While some work shows that multi-task networks trained via unitary scalarization exhibit superior performance to independent per-task models [29, 35], others suggest the opposite [30, 54, 58].

Original: As a result, many explanations for the difﬁculty of MTL have been proposed, each motivating anew Specialized Multi-Task Optimizer (SMTO) 
Revised: As a result many explanations for the difﬁculty of MTL have been proposed, each motivating a new Specialized Multi-Task Optimizer (SMTO) [11, 42, 54, 62, 66].

Original: These works typically claim thatthe proposed SMTO outperforms unitary scalarization, in addition to relevant prior work.
Revised: These works typically claim that the proposed SMTO outperforms unitary scalarization, in addition to relevant prior work.

Original: However,
Revised: However,

Original: SMTOs usually require access to per-task gradients either with respect to the shared parameters,or to the shared representation.
Revised: SMTOs usually require access to per-task gradients either with respect to the shared parameters, or to the shared representation.

Original: Therefore, their reported performance gain comes at signiﬁcantcomputation and memory cost, the overhead scaling linearly with the number of tasks.
Revised: Therefore, their reported performance gain comes at signiﬁcant computation and memory cost, the overhead scaling linearly with the number of tasks.

Original: By contrast,unitary scalarization requires only the average of the gradients across tasks, which can be computedvia a single backpropagation.
Revised: By contrast, unitary scalarization requires only the average of the gradients across tasks, which can be computed via a single backpropagation.

Original: Existing SMTOs were introduced to solve challenges related to the optimization of the deep MTLproblem.
Revised: Existing SMTOs were introduced to solve challenges related to the optimization of the deep MTL problem.

Original: We instead postulate that the reported weakness of unitary scalarization is linked toexperimental variability or to a lack of regularization, leading to the following contributions:
Revised: We instead postulate that the reported weakness of unitary scalarization is linked to experimental variability or to a lack of regularization, leading to the following contributions:

Revised: ∗ Equal contribution.

Original: • A comprehensive experimental evaluation ( § 4) of recent SMTOs on popular multi-task benchmarks, showing that no SMTO consistently outperforms unitary scalarization in spite of the added complexity and overhead.
Revised: • A comprehensive experimental evaluation ( § 4) of recent SMTOs on popular multi-task benchmarks, showing that no SMTO consistently outperforms unitary scalarization in spite of the added complexity and overhead.

Original: In particular, either the differences between unitary scalarization and SMTOs are not statistically signiﬁcant, or they can be bridged by standard regularization and stabilization techniques from the single-task literature.
Revised: In particular, either the differences between unitary scalarization and SMTOs are not statistically signiﬁcant, or they can be bridged by standard regularization and stabilization techniques from the single-task literature.

Original: Our reinforcement learning (RL)experiments include optimizers previously applied only to supervised learning.•
Revised: Our reinforcement learning (RL) experiments include optimizers previously applied only to supervised learning.

Original:  An empirical and technical analysis of the considered SMTOs, suggesting that they reduceoverﬁtting on the multi-task problem and hence act as regularizers ( § 5).
Revised: • An empirical and technical analysis of the considered SMTOs, suggesting that they reduce overﬁtting on the multi-task problem and hence act as regularizers ( § 5).

Original: We conduct an ablationstudy and provide a collection of novel and existing technical results that support this hypothesis.•
Revised: We conduct an ablation study and provide a collection of novel and existing technical results that support this hypothesis.

Original: We will publicly release a uniﬁed PyTorch [49] codebase, currently provided in the supplementarymaterial, for all SMTOs in the experimental evaluation.
Revised: • Code to reproduce the experiments, including a uniﬁed PyTorch [50] implementation of the considered SMTOs, is available at https://github.com/yobibyte/ unitary-scalarization-dmtl .

Original: We believe that our results suggest that the considered SMTOs can be often replaced by lessexpensive techniques.
Revised: We believe that our results suggest that the considered SMTOs can be often replaced by less expensive techniques.

Original: We hope that these surprising results stimulate the search for a deeper understanding of MTL
Revised: We hope that these surprising results stimulate the search for a deeper understanding of MTL.

Original:  Related Work
Revised:  Related Work

Original: Before diving into details of speciﬁc SMTOs in Section 5, we provide a high-level overview of thedeep MTL research.
Revised: Before diving into details of speciﬁc SMTOs in Section 5, we provide a high-level overview of the deep MTL research.

Original: MTL is represented by a vast body of literature, including Caruana’s pioneeringwork [6] exploring hard parameter sharing , i.e., sharing neural network parameters between all taskswith, possibly, a separate part of the model for each task.
Revised: MTL is represented by a vast body of literature, including Caruana’s pioneering work [6] exploring hard parameter sharing , i.e., sharing neural network parameters between all tasks with, possibly, a separate part of the model for each task.

Original: Hard parameter sharing is still the major
Revised: Hard parameter sharing is still the major

Original: MTL approach adopted in natural language processing [9, 12], computer vision [45], and speechrecognition [52].
Revised: MTL approach adopted in natural language processing [9, 12], computer vision [46], and speech recognition [53].

Original: In this work, we implicitly assume that each parameter update employs informationfrom all tasks.
Revised: In this work, we implicitly assume that each parameter update employs information from all tasks.

Original: However, not all works satisfy this assumption, either due to a large number oftasks [4, 35], or simply as an implementation decision [25, 36].
Revised: However, not all works satisfy this assumption, either due to a large number of tasks [4, 36], or simply as an implementation decision [25, 37].

Original: In this setting, MTL resembles otherproblems dealing with multiple tasks, i.e., continual [31], curriculum [46], and meta-learning [24],which are not the focus of this work.
Revised: In this setting, MTL resembles other problems dealing with multiple tasks, i.e., continual [32], curriculum [47], and meta-learning [24], which are not the focus of this work

Original: Many works strive to improve the performance of deep multi-task models.
Revised: Many works strive to improve the performance of deep multi-task models.

Original: One line of researchhypothesizes that conﬂicting per-task gradient directions lead to suboptimal models, and thus focuseson explicitly removing conﬂicts across task gradients [11, 27, 40, 41, 61, 64].
Revised: One line of research hypothesizes that conﬂicting per-task gradient directions lead to suboptimal models, and thus focuses on explicitly removing conﬂicts across task gradients [11, 28, 41, 42, 62, 66].

Original: Some authors postulatethat loss imbalances across tasks hinder learning, proposing loss reweighting methods [10, 29, 39].
Revised: Some authors postulate that loss imbalances across tasks hinder learning, proposing loss reweighting methods [10, 30, 40].

Original: Finally, Sener and Koltun [53] and Navon et al.
Revised: Finally, Sener and Koltun [54] and Navon et al.

Original: [47] propose that tasks compete for model capacityand interpret MTL as multi-objective optimization in order to cope with inter-task competition.
Revised: [48] propose that tasks compete for model capacity and interpret MTL as multi-objective optimization in order to cope with inter-task competition.

Original: Here, we focus on algorithms that explicitly rely on per-task gradients to try to outperform unitaryscalarization ( § 5).
Revised: Here, we focus on algorithms that explicitly rely on per-task gradients to try to outperform unitary scalarization ( § 5).

Original: Research on multi-task architectures [19, 45] or MTL algorithms exclusivelymotivated by deterministic loss reweighting [18, 29, 42] are orthogonal to our work.
Revised: Research on multi-task architectures [19, 46] or MTL algorithms exclusively motivated by deterministic loss reweighting [18, 30, 43] are orthogonal to our work.

Original: Both topics areinvestigated by a recent survey on pixel-level multi-task computer vision problems [60], which foundthat the minimization of tuned weighted sums of losses  is empirically competitive with deterministicloss reweighting and MGDA in the considered settings.
Revised: Both topics are investigated by a recent survey on pixel-level multi-task computer vision problems [61], which found that the minimization of tuned weighted sums of losses (scalarizations) is empirically competitive with deterministic loss reweighting and MGDA in the considered settings.

Revised: These results are extended to popular SMTOs by a critical review from Xin et al.

Revised: [63], which is concurrent to our work (published at the same venue).

Revised: The study argues that the performance proﬁle of SMTOs can be recovered by tuning scalarization coefﬁcients, which appear to play an important role in both optimization and generalization.

Revised: Our work, instead, provides a complementary perspective by focusing on the unitary scalarization and on the comparatively regularizing effect of SMTOs.

Original: In addition to the common supervised settings, we also consider multi-task RL, whose research canbe grouped into three categories: the ﬁrst adds auxiliary tasks providing additional inductive biases tospeed up learning [26] on a target task.
Revised: In addition to the common supervised settings, we also consider multi-task RL, whose research can be grouped into three categories: the ﬁrst adds auxiliary tasks providing additional inductive biases to speed up learning [27] on a target task.

Original: The second, based on policy distillation, uses per-task teachermodels to provide labels for a multi-task model or per-task policies as regularizers [48, 50, 56].
Revised: The second, based on policy distillation, uses per-task teacher models to provide labels for a multi-task model or per-task policies as regularizers [49, 51, 57].

Original: Thethird directly learns a shared policy [28], possibly via an SMTO [64].
Revised: The third directly learns a shared policy [29], possibly via an SMTO [66].

Original: We focus on the third category,whose literature reports varying performance for unitary scalarization (better [28] or worse [64] thanper-task models), indicating confounding factors in evaluation pipelines and further motivating ourwork.
Revised: We focus on the third category, whose literature reports varying performance for unitary scalarization (better [29] or worse [66] than per-task models), indicating confounding factors in evaluation pipelines and further motivating our work.

Original: PopArt [23, 59] performs scale-invariant value function updates in order to address differencesin returns across environments, showing improvements in the multi-task setting while still usingunitary scalarization.
Revised: PopArt [23, 60] performs scale-invariant value function updates in order to address differences in returns across environments, showing improvements in the multi-task setting while still using unitary scalarization.

Original: PopArt does not require per-task gradients but introduces additional hyperparameters.
Revised: PopArt does not require per-task gradients but introduces additional hyperparameters.

Original: In our work, we address the differences in rewards by normalizing them at the replay bufferlevel.
Revised: In our work, we address the differences in rewards by normalizing them at the replay buffer level.

Original: However, we believe both unitary scalarization and SMTOs might equally beneﬁt from PopArt.
Revised: However, we believe both unitary scalarization and SMTOs might equally beneﬁt from PopArt.

Original:  Multi-Task Learning Optimizers
Revised:  Multi-Task Learning Optimizers

Original: We will now describe the deep MTL training problem and common algorithms employed for itssolution.
Revised: We will now describe the deep MTL training problem and popular algorithms employed for its solution.

Original: Let ( X, Y ) ∈ R
Revised: Let ( X, Y ) ∈ R

Original: d × n × R o × n be the training set, composed of n d -dimensional points ando -dimensional labels.
Revised: d × n × R o × n be the training set, composed of n d -dimensional points and o -dimensional labels.

Original: In addition, L i : R o × n × R o × n → R denotes the loss for the i -th task, θ ∈ R Sthe parameter space, T := { 1 , . . .
Revised: In addition, L i : R o × n × R o × n → R denotes the loss for the i -th task, θ ∈ R S the parameter space, T := { 1 , . . .

Original: , m } the set of m tasks.
Revised: , m } the set of m tasks.

Original: The goal of MTL is to learn a single(generally task-aware) parametrized model f : R S × R d × n × T → R o × n that performs well on all tasks T .
Revised: The goal of MTL is to learn a single (generally task-aware) parametrized model f : R S × R d × n × T → R o × n that performs well on all tasks T .

Original: The parameter space is often split into a set of shared parameters across tasks (generally themajority of the architecture), denoted θ (cid:107) , and (possibly empty) task-speciﬁc parameters, denoted θ ⊥ ,so that θ :=
Revised: The parameter space is often split into a set of shared parameters across tasks (generally the majority of the architecture), denoted θ (cid:107) , and (possibly empty) task-speciﬁc parameters, denoted θ ⊥ , so that θ :=

Original: [ θ (cid:107) , θ ⊥ ] T .
Revised: [ θ (cid:107) , θ ⊥ ] T .

Original: In this context, the model f often takes on an encoder-decoder architecture,where the encoder g learns a shared representation across tasks, and the decoders h
Revised: In this context, the model f often takes on an encoder-decoder architecture, where the encoder g learns a shared representation across tasks, and the decoders h

Original: i are task-speciﬁcpredictive heads: f ( θ , X, i )
Revised: i are task-speciﬁc predictive heads: f ( θ , X, i )

Original: =
Revised: =

Original: h i ( g ( θ (cid:107) , X ) , θ ⊥ ) .
Revised: h i ( g ( θ (cid:107) , X ) , θ ⊥ ) .

Original: In this case, we denote by z = g ( θ (cid:107) , X ) ∈ R r × nthe r -dimensional shared representation of X .
Revised: In this case, we denote by z = g ( θ (cid:107) , X ) ∈ R r × n the r -dimensional shared representation of X .

Original: The training problem for MTL is typically formulated as the sum of the per-task losses [11, 53, 64]:
Revised: The training problem for MTL is typically formulated as the sum of the per-task losses [11, 54, 66]:

Original: [Equation]
Revised: [Equation]

Original: Unitary Scalarization
Revised: Unitary Scalarization

Original: The obvious way to minimize the multi-task training objective in equation(1) is to rely on a standard gradient-based algorithm.
Revised: The obvious way to minimize the multi-task training objective in equation (1) is to rely on a standard gradient-based algorithm.

Original: While, for simplicity, we focus on standardgradient descent rather than mini-batch stochastic gradient descent, the notation can be adapted byreplacing the dataset size n by the mini-batch size b .
Revised: While, for simplicity, we focus on standard gradient descent rather than mini-batch stochastic gradient descent, the notation can be adapted by replacing the dataset size n by the mini-batch size b .

Original: Equation (1) corresponds to a linear scalarizationwith unitary weights under a multi-objective interpretation of MTL; hence, we call the directapplication of gradient descent on equation (1) unitary scalarization .
Revised: Equation (1) corresponds to a linear scalarization with unitary weights under a multi-objective interpretation of MTL; hence, we call the direct application of gradient descent on equation (1) unitary scalarization .

Original: For vanilla gradient descent, thiscorresponds to taking a step in the opposite direction as the one given by the sum of per-task gradients:∇ θ L MT =
Revised: For vanilla gradient descent, this corresponds to taking a step in the opposite direction as the one given by the sum of per-task gradients: ∇ θ L MT =

Original: (cid:80) i ∈T ∇ θ L i .
Revised: (cid:80) i ∈T ∇ θ L i .

Original: Per-task gradients are not required, as it sufﬁces to directly compute thegradient of the sum L MT .
Revised: Per-task gradients are not required, as it sufﬁces to directly compute the gradient of the sum L MT .

Original: Hence, when relying on deep learning frameworks based on reverse-modedifferentiation, such as PyTorch [49], the backward pass is performed once per iteration (rather than[Figure] m times).
Revised: Hence, when relying on deep learning frameworks based on reverse-mode differentiation, such as PyTorch [50], the backward pass is performed once per iteration (rather than m times).

Original: Furthermore, the memory cost is a factor m less than most SMTOs, which require accessto each ∇ θ L i .
Revised: Furthermore, the memory cost is a factor m less than most SMTOs, which require access to each ∇ θ L i .

Original: As a consequence, unitary scalarization is simple, fast, and memory efﬁcient.
Revised: As a consequence, unitary scalarization is simple, fast, and memory efﬁcient.

Original: Ourexperiments demonstrate that, when possibly coupled with single-task regularization such as earlystopping, (cid:96) 2 penalty or dropout layers [55], this simple optimizer is strongly competitive with SMTOs.
Revised: Our experiments demonstrate that, when possibly coupled with single-task regularization such as early stopping, (cid:96) 2 penalty or dropout layers [56], this simple optimizer is strongly competitive with SMTOs.

Original: MGDA Sener and Koltun [53] point out that equation (1) can be cast as a multi-objective optimization problem with the following objective: L MT ( θ ) :=
Revised: MGDA Sener and Koltun [54] point out that equation (1) can be cast as a multi-objective optimization problem with the following objective: L MT ( θ ) :=

Original: [ L 1 ( θ ) , . . .
Revised: [ L 1 ( θ ) , . . .

Original: , L m ( θ )] T .
Revised: , L m ( θ )] T .

Original: A commonlyemployed solution concept in multi-objective optimization is Pareto optimality.
Revised: A commonly employed solution concept in multi-objective optimization is Pareto optimality.

Original: A point θ ∗ iscalled Pareto-optimal if, for any another point θ † such that ∃ i ∈ T : L i ( θ † ) <
Revised: A point θ ∗ is called Pareto-optimal if, for any another point θ † such that ∃ i ∈ T : L i ( θ † ) <

Original: L i ( θ ∗ ) , then∃ j ∈ T : L j ( θ † ) >
Revised: L i ( θ ∗ ) , then ∃ j ∈ T : L j ( θ † ) >

Original: L j ( θ ∗ ) .
Revised: L j ( θ ∗ ) .

Revised: A necessary condition for Pareto optimality at a point is Pareto stationarity, deﬁned as the lack of a shared descent direction across all losses at that point.

Original: Sener and Koltun [53] rely on Multiple-Gradient Descent Algorithm(MGDA) [14] to reach a Pareto-stationary point for shared parameters θ (cid:107) .
Revised: Sener and Koltun [54] rely on Multiple-Gradient Descent Algorithm (MGDA) [14] to reach a Pareto-stationary point for shared parameters θ (cid:107) .

Original:  MGDA takes steps ina directionthat decreases the loss of all tasks at once [14, 17], which can be found by solving thefollowing optimization problem:
Revised: Intuitively, MGDA proceeds by repeatedly stepping in a shared descent direction [14, 17], which can be found by solving the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: whose dual takes the following form (corresponding to the formulation from Désidéri [14]):
Revised: whose dual takes the following form (corresponding to the formulation from Désidéri [14]):

Original: [Equation]
Revised: [Equation]

Original: In other words, MGDA takes a step in a direction g given by the negative convex combinationof per-task gradients, whose coefﬁcients are given by solving equation (3).
Revised: In other words, MGDA takes a step in a direction g given by the negative convex combination of per-task gradients, whose coefﬁcients are given by solving equation (3).

Original: In practice, per-taskgradients are rescaled before applying MGDA:
Revised: In practice, per-task gradients are rescaled before applying MGDA:

Original: the original authors’ implementation [53] relieson ∇ θ (cid:107) L i ← ∇ θ (cid:107) L i / (cid:13)(cid:13)(cid:13) ∇ θ (cid:107) L i (cid:13)(cid:13)(cid:13)
Revised: the original authors’ implementation [54] relies on ∇ θ (cid:107) L i ← ∇ θ (cid:107) L i / (cid:13)(cid:13)(cid:13) ∇ θ (cid:107) L i (cid:13)(cid:13)(cid:13)

Original: L i ( θ ) .
Revised: L i ( θ ) .

Original: The convergence of MGDA to a Pareto-optimal point is stillguaranteed after normalization [14].
Revised: The convergence of MGDA to a Pareto-stationary point is still guaranteed after normalization [14].

Original: IMTL Impartial Multi-Task Learning (IMTL) [41] is presented as an SMTO that is not biasedagainst any single task.
Revised: IMTL Impartial Multi-Task Learning (IMTL) [42] is presented as an SMTO that is not biased against any single task.

Original: It is composed of two complementary algorithmic blocks: IMTL-L, acting ontask losses, and IMTL-G, acting on per-task gradients.
Revised: It is composed of two complementary algorithmic blocks: IMTL-L, acting on task losses, and IMTL-G, acting on per-task gradients.

Original: IMTL-G follows the intuition that a multi-taskoptimizer should proceed along a direction g = − (cid:80) i
Revised: IMTL-G follows the intuition that a multi-task optimizer should proceed along a direction g = − (cid:80) i

Original: α i ∇ θ (cid:107) L i that equally represents per-taskgradients.
Revised: α i ∇ θ (cid:107) L i that equally represents per-task gradients.

Original: This is formulated analytically by requiring that the cosine similarity between g and each∇ θ (cid:107) L i be the same.
Revised: This is formulated analytically by requiring that the cosine similarity between g and each ∇ θ (cid:107) L i be the same.

Original: To prevent the resulting problem from being underdetermined, Liu et al.
Revised: To prevent the resulting problem from being underdetermined, Liu et al.

Original: [41]add the constraint (cid:80) i ∈T α i = 1 , resulting in a problem that admits a closed-form solution for g : 
Revised: [42] add the constraint (cid:80) i ∈T α i = 1 , resulting in a problem that admits a closed-form solution for g : [Equation] T

Original: [Equation]
Revised: [Equation]

Original: IMTL-L, instead, aims to reweight task losses so that they are all constant over time, and equalto 1 .
Revised: IMTL-L, instead, aims to reweight task losses so that they are all constant over time, and equal to 1 .

Original: In order to limit oscillations of the scaling factors, the authors propose to learn them jointly with the network by minimizing a common objective via gradient descent.
Revised: In order to limit oscillations of the scaling factors, the authors propose to learn them jointly with the network by minimizing a common objective via gradient descent.

Original: In particular, givens i ∈ R ∀ i ∈ T , Liu et al.
Revised: In particular, given s i ∈ R ∀ i ∈ T , Liu et al.

Original: [41] derive the following form for the joint minimization problem:min s , θ [ (cid:80) i ( e s i
Revised: [42] derive the following form for the joint minimization problem: min s , θ [ (cid:80) i ( e s i

Original: L i ( θ ) − s i )] .
Revised: L i ( θ ) − s i )] .

Original: As proved by Liu et al.
Revised: As proved by Liu et al.

Original: [41], IMTL-L only has a rescaling effect onthe update direction of IMTL-G.
Revised: [42], IMTL-L only has a rescaling effect on the update direction of IMTL-G.

Original: Unlike IMTL-G and the other SMTOs presented in this section,
Revised: Unlike IMTL-G and the other SMTOs presented in this section,

Original: IMTL-L rescaling is designed to affect the updates for task-speciﬁc parameters θ ⊥ as well.
Revised: IMTL-L rescaling is designed to affect the updates for task-speciﬁc parameters θ ⊥ as well.

Original: PCGrad Let us write cos( x , z ) for the cosine similarity between vectors x and z .
Revised: PCGrad Let us write cos( x , z ) for the cosine similarity between vectors x and z .

Original: Yu et al.
Revised: Yu et al.

Original: [64]postulate that multi-task convergence is severely slowed down if the following three conditions(named the tragic triad ) hold at once: (i) conﬂicting gradient directions: cos( ∇θ (cid:107) L i , ∇ θ (cid:107) L j ) <
Revised: [66] postulate that multi-task convergence is severely slowed down if the following three conditions (named the tragic triad ) hold at once: (i) conﬂicting gradient directions: cos( ∇

Original: 0for some i, j ∈ T ; (ii) differing gradient magnitudes: (cid:13)(cid:13) ∇ θ (cid:107) L i (cid:13)(cid:13)
Revised: [Equation] <for some i, j ∈ T ; (ii) differing gradient magnitudes: (cid:13) (cid:13) ∇ θ [Equation]

Original:  (cid:29) (cid:13)(cid:13) ∇ θ (cid:107) L j (cid:13)(cid:13) for some i, j ∈ T ; and(iii) the unitary scalarization L MT has high curvature along ∇ θ (cid:107) L MT .
Revised: L [Equation] (cid:13) (cid:29) (cid:13) (cid:13) ∇ θ (cid:107) L j (cid:13) (cid:13) for some i, j ∈ T ; and (iii) the unitary scalarization L MT has high curvature along ∇ θ (cid:107) L MT .

Original: The PCGrad [64] SMTO ispresented as a solution to the tragic triad, targeted at the ﬁrst condition.
Revised: The PCGrad [66] SMTO is presented as a solution to the tragic triad, targeted at the ﬁrst condition.

Original: Consistent with the previoussections, let us denote the update direction by g .
Revised: Consistent with the previous sections, let us denote the update direction by g .

Original: Furthermore, let [ x ] + := max( x , 0 ) .
Revised: Furthermore, let [ x ] + := max( x , 0 ) .

Original: Given per-taskgradients ∇ θ (cid:107) L i , PCGrad iteratively projects each task gradient onto the normal plane of all thegradients with which it conﬂicts:
Revised: Given per-task gradients ∇ θ (cid:107) L i , PCGrad iteratively projects each task gradient onto the normal plane of all the gradients with which it conﬂicts:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: x ) (cid:13)(cid:13)(cid:13)

Original: [Table] (cid:107) L j (cid:13)(cid:13)(cid:13)(cid:35) + ∇ θ (cid:107) L j ∀ j ∈ T \ { i } (cid:35) ∀ i ∈ T , g = − (cid:88) i ∈T g i , (5) where the iterative updates of g i with respect to ∇ θ (cid:107) L j are performed in random order.
Revised: where the iterative updates of g i with respect to ∇ θ (cid:107) L j are performed in random order.

Original: GradDrop Chen et al.
Revised: GradDrop Chen et al.

Original: [11] focus on conﬂicting signs across task gradient entries, arguing thatsuch conﬂicts lead to gradient “tug-of-wars".
Revised: [11] focus on conﬂicting signs across task gradient entries, arguing that such conﬂicts lead to gradient “tug-of-wars".

Original: The GradDrop SMTO [11], presented as a solution tothis problem, proposes to randomly mask per-task gradients ∇ θ (cid:107)
Revised: The GradDrop SMTO [11], presented as a solution to this problem, proposes to randomly mask per-task gradients ∇ θ (cid:107)

Original: L
Revised: L

Original: i so as to minimize such conﬂicts.
Revised: i so as to minimize such conﬂicts.

Original: Speciﬁcally, GradDrop computes the “positive sign purity" p j for the task gradient’s j -th entry andthen masks the j -th entry of each per-task gradient with probability increasing with p j , if the entry isnegative, or decreasing with p j , if the entry is positive.
Revised: Speciﬁcally, GradDrop computes the “positive sign purity" p j for the task gradient’s j -th entry and then masks the j -th entry of each per-task gradient with probability increasing with p j , if the entry is negative, or decreasing with p j , if the entry is positive.

Original: Let us write p :=
Revised: Let us write p :=

Original: [ p 1 , . .
Revised: [ p 1 , . .

Original: . , p S ] , where S is thedimensionality of the parameter space (see § 3), (cid:12) for the Hadamard product and 1 a for the indicatorvector on condition a .
Revised: . , p S ] , where S is the dimensionality of the parameter space (see § 3), (cid:12) for the Hadamard product and 1 a for the indicator vector on condition a .

Original: Given a vector u i , uniformly sampled in [ 0 , 1 ] at each iteration, GradDroptakes a step in the direction given by:
Revised: Given a vector u i , uniformly sampled in [ 0 , 1 ] at each iteration, GradDrop takes a step in the direction given by:

Original: [Equation]
Revised: [Equation]

Original: ) 

Original:  , with p = 1(cid:32) 1 + (cid:80) i ∈T ∇ θ (cid:107) L i (cid:80) i ∈T (cid:12)(cid:12) ∇ θ (cid:107) L i (cid:12)(cid:12) (cid:33) .

Original:  Experimental Evaluation
Revised:  Experimental Evaluation

Original: Relying on a uniﬁed experimental pipeline, we present an empirical evaluation on common MTLbenchmarks of unitary scalarization ( § 3), of the popular SMTOs presented in § 3, and of the recent
Revised: Relying on a uniﬁed experimental pipeline, we present an empirical evaluation on common MTL benchmarks of unitary scalarization ( § 3), of the popular SMTOs presented in § 3, and of the recent

Original: RLW algorithms [39] due to their similarities with PCGrad and GradDrop (see § 5.2).
Revised: RLW algorithms [40] due to their similarities with PCGrad and GradDrop (see § 5.2).

Original: In particular,we benchmark against the two RLW instances that showed the best average performance in theoriginal paper: RLW with weights sampled from a Dirichlet distribution (“RLW Diri.”), and RLWwith weights sampled from a Normal distribution (“RLW Norm.”).
Revised: We benchmark against the two RLW instances that showed the best average performance in the original paper: RLW with weights sampled from a Dirichlet distribution (“RLW Diri.”), and RLW with weights sampled from a Normal distribution (“RLW Norm.”).

Revised: The goal of this section is to assess the efﬁcacy of a popular line of previous work, focusing on a few representative or well-established optimizers.

Revised: Therefore, we forego comparison with more recent SMTOs [28, 41, 48] as they have not yet encountered widespread adoption, often lack signiﬁcant enough improvements over the optimizers we consider, or may have substantial commonalities with them (see § 5.2 for Nash-MTL [48], which was published concurrently to the ﬁnalization of this work).

Original: Whenever appropriate, we employ“Unit.
Revised: Whenever appropriate, we employ “Unit.

Original: Scal.”
Revised: Scal.”

Original: as shorthand for unitary scalarization.
Revised: as shorthand for unitary scalarization.

Original: We ﬁrst present supervised learning experiments( § 4.1), and then evaluate on a popular reinforcement learning benchmark ( § 4.2).
Revised: We ﬁrst present supervised learning experiments ( § 4.1), and then evaluate on a popular reinforcement learning benchmark ( § 4.2).

Original: Our experiments indicate that the performance of unitary scalarization has been consistently underestimated in the literature.

Original: By showing the variability between runs and by relying on standardregularization and stabilization techniques from the single-task literature, we demonstrate that no

Original: SMTO consistently outperforms unitary scalarization across the considered settings .

Original: This resultholds in spite of the added complexity and computational overhead associated with most SMTOs.

Original: Furthermore, in supervised learning, most methods drive the training loss of all tasks in the proximityof the respective global minima.

Original: This suggests that the main difﬁculty of MTL is not associated withthe optimization of its training objective, but rather to incorporating adequate regularization (cf.

Original: 4.1 Supervised Learning

Original: All the architectures employed in the supervised learning experiments conform to the encoder-decoderstructure detailed in § 3.

Original: Whenever suggested by the original authors for this context, the SMTO

Original: Unit.
Revised: Unit.

Original: Scal.
Revised: Scal.

Original: IMTL
Revised: IMTL

Original: MGDA
Revised: MGDA

Original: GradDrop
Revised: GradDrop

Original: PCGrad
Revised: PCGrad

Original: RLW Diri.
Revised: RLW Diri.

Original: RLW Norm.
Revised: RLW Norm.

Original: A v e r a g e T a s k A cc u r a c
Revised: A v e r a g e T a s k A cc u r a c

Original: y
Revised: y

Original: (a) Avg. task test accuracy: mean and 95 % CI (10 runs).
Revised: (a) Avg. task test accuracy: mean and 95 % CI (10 runs).

Original: Unit.
Revised: Unit.

Original: Scal.
Revised: Scal.

Original: IMTL
Revised: IMTL

Original: MGDA
Revised: MGDA

Original: GradDrop
Revised: GradDrop

Original: PCGrad
Revised: PCGrad

Original: RLW Diri.
Revised: RLW Diri.

Original: RLW Norm.
Revised: RLW Norm.

Original: E p o c h r un t i m e
Revised: E p o c h r un t i m e

Original: [ s ] implementations rely on per-task gradients with respect to the last shared activation, ∇ z , ratherthan on  the usually more expensive ∇ θ L i .
Revised: [ s ] (b) Box plots Our experiments indicate that the performance of unitary scalarization has been consistently underestimated in the literature. By showing the variability between runs and by relying on standard regularization and stabilization techniques from the single-task literature, we demonstrate that no SMTO consistently outperforms unitary scalarization across the considered settings .

Revised: This result holds in spite of the added complexity and computational overhead associated with most SMTOs.

Revised: Furthermore, in supervised learning, most methods drive the training loss of all tasks in the proximity of the respective global minima.

Revised: This suggests that the main difﬁculty of MTL is not associated with the optimization of its training objective, but rather to incorporating adequate regularization (cf.

Revised: 4.1 Supervised Learning

Revised: All the architectures employed in the supervised learning experiments conform to the encoder-decoder structure detailed in § 3.

Revised: Whenever suggested by the original authors for this context, the SMTO implementations rely on per-task gradients with respect to the last shared activation, ∇ z , rather than on the usually more expensive ∇ θ L i .

Original: In particular, this is the case for MGDA, IMTL and
Revised: In particular, this is the case for MGDA, IMTL and

Original: GradDrop.
Revised: GradDrop.

Original: See appendix B for details concerning each individual algorithm.
Revised: See appendix B for details concerning each individual algorithm.

Original: Surprisingly, several
Revised: Surprisingly, several

Original: MTL works [11, 39, 41, 64] report validation results, making it easier to overﬁt.
Revised: MTL works [11, 40, 42, 66] report validation results, making it easier to overﬁt.

Original: Instead, followingstandard machine learning practice, we select a model on the validation set, and later report testmetrics for all benchmarks.
Revised: Instead, following standard machine learning practice, we select a model on the validation set, and later report test metrics for all benchmarks.

Original: Validation results are also available in appendix D. Appendix C.1 reportsdataset descriptions, the computational setup, hyperparameter and tuning details.
Revised: Validation results are also available in appendix D. Appendix C.1 reports dataset descriptions, the computational setup, hyperparameter and tuning details.

Original: 4.1.1 Multi-MNIST
Revised: 4.1.1 Multi-MNIST

Original: We present results on the Multi-MNIST [53] dataset, a simple two-task supervised learning benchmark.
Revised: We present results on the Multi-MNIST [54] dataset, a simple two-task supervised learning benchmark.

Original: We employ a popular architecture from previous work [53, 64] (see appendix C.1), where a singledropout layer [55] (with dropout probability 0 . 5 ) is employed in both the encoder and the decoder.
Revised: We employ a popular architecture from previous work [54, 66] (see appendix C.1), where a single dropout layer [56] (with dropout probability 0 . 5 ) is employed in both the encoder and the decoder.

Original: (cid:96) 2regularization did not improve validation performance and was therefore omitted.
Revised: (cid:96)regularization did not improve validation performance and was therefore omitted.

Original: Figure 1 reports theaverage task test accuracy, and the training time per epoch.
Revised: Figure 1 reports the average task test accuracy, and the training time per epoch.

Original: For each run, the test model was selectedas the model with the largest average task validation accuracy across the training epochs.
Revised: For each run, the test model was selected as the model with the largest average task validation accuracy across the training epochs.

Original: Appendix Dpresents the results of Figure 1 in tabular form, as well as the average task validation accuracy perepoch.
Revised: Appendix D presents the results of Figure 1 in tabular form, as well as the average task validation accuracy per epoch.

Original: As seen from the overlapping conﬁdence intervals, none of the considered algorithms clearlyoutperforms the others.
Revised: As seen from the overlapping conﬁdence intervals, none of the considered algorithms clearly outperforms the others.

Original: However, GradDrop displays higher experimental variability.
Revised: However, GradDrop displays higher experimental variability.

Original: Furthermore,
Revised: Furthermore,

Original: Figure 6(b) shows that the sums of the task cross-entropy losses is driven nearly to zero by most methods.
Revised: Figure 7(b) shows that the sums of the task cross-entropy losses is driven nearly to zero by most methods.

Original: Finally, Figure 1(b) shows that unitary scalarization also has among the lowest training times.
Revised: Finally, Figure 1(b) shows that unitary scalarization also has among the lowest training times.

Original: 4.1.2 CelebA
Revised: 4.1.2 CelebA

Original: We now show results for the CelebA [43] dataset, a challenging 40 -task multi-label classiﬁcation problem.
Revised: We now show results for the CelebA [44] dataset, a challenging 40 -task multi-label classiﬁcation problem.

Original: We employ the same architecture as many previous studies [39, 41, 53, 64] (see appendix C.1).
Revised: We employ the same architecture as many previous studies [40, 42, 54, 66] (see appendix C.1).

Original: We tuned (cid:96) 2 regularization terms λ for all SMTOs in the following grid: λ ∈ { 0 , 10 − 4 , 10 − 3 } .
Revised: We tuned (cid:96) 2 regularization terms λ for all SMTOs in the following grid: λ ∈ { 0 , 10 − 4 , 10 − 3 } .

Original: Thebest validation performance was attained with λ = 10 − 3 for unitary scalarization, IMTL and PCGrad,and with λ = 10 − 4 for MGDA, GradDrop, and RLW.
Revised: The best validation performance was attained with λ = 10 − 3 for unitary scalarization, IMTL and PCGrad, and with λ = 10 − 4 for MGDA, GradDrop, and RLW.

Original: Validation performance was further stabilizedby the addition of several dropout layers (see Figure 5), with dropout probabilities from 0 .
Revised: Validation performance was further stabilized [Figure] A v e r a g e T a s k A cc u r a c y Unit.

Revised: Scal.

Revised: IMTL

Revised: MGDA

Revised: GradDrop

Revised: PCGrad

Revised: RLW Diri.

Revised: RLW Norm.

Revised: E p o c h r un t i m e

Revised: [ s ] (

Revised: [Figure]

Revised: [Figure]

Revised: by the addition of several dropout layers (see Figure 5), with dropout probabilities from 0 .

Original: 25 to 0 .
Revised: 25 to 0 .

Original: We present an ablation study on the effect of regularization on this experiment in § 5.1.
Revised: We present an ablation study on the effect of regularization on this experiment in § 5.1.

Original: Figure 9(appendix D.2) shows that regularization improves the peak average validation performance for allthe considered methods.
Revised: Figure(appendix D.2) shows that regularization improves the peak average validation performance for all the considered methods.

Original: Analogously to our Multi-MNIST results, Figure 2 plots the distribution ofthe training time per epoch, and the average test task accuracy.
Revised: Analogously to our Multi-MNIST results, Figure 2 plots the distribution of the training time per epoch, and the average test task accuracy.

Original: As with Multi-MNIST, the test modelfor each run was the one with maximal average validation task accuracy across epochs.
Revised: As with Multi-MNIST, the test model for each run was the one with maximal average validation task accuracy across epochs.

Original: In otherwords, if the peak is attained before the last epoch, we perform early stopping: as shown in Figure7(a) in appendix D this is the case for most methods.
Revised: In other words, if the peak is attained before the last epoch, we perform early stopping: as shown in Figure 8(a) in appendix D this is the case for most methods.

Original: Due to the large number of tasks, Figure 2(b)shows relatively large runtime differences across methods.
Revised: Due to the large number of tasks, Figure 2(b) shows relatively large runtime differences across methods.

Original: PCGrad is the slowest (roughly 35 timesslower than unitary scalarization).
Revised: PCGrad is the slowest (roughly 35 times slower than unitary scalarization).

Original: In fact, amongst the considered algorithms, it is the only one thatcomputes per-task gradients over the parameters ( ∇ θ L i ∀ i ∈ T ) at each iteration.
Revised: In fact, amongst the considered algorithms, it is the only one that computes per-task gradients over the parameters ( ∇ θ L i ∀ i ∈ T ) at each iteration.

Original: GradDrop, MGDAand IMTL have overhead factors (compared to unitary scalarization) ranging from roughly 1 .[-
 to [Figure] v-]
Revised: GradDrop, MGDA and IMTL have overhead factors (compared to unitary scalarization) ranging from roughly 1 .[-
 to [Figure] v-]

Original: [Figure]

Original: r a g e T a

Original: [Figure]

Original: k A cc u r a c y

Original: [Figure] [Figure] [Figure] which proves to be the most cost-effective
Revised: 05 to 2 .

Original: algorithm

Original: 4 due to the relatively small size of z for the employed architecture.
Revised: 4 due to the relatively small size of z for the employed architecture.

Original: The overhead of RLW isnegligible: roughly 5% .
Revised: The overhead of RLW is negligible: roughly 5% .

Original: Nevertheless, due to largely overlapping conﬁdence intervals in Figure 2(a),none of the methods consistently outperforms unitary scalarization.
Revised: Nevertheless, due to largely overlapping conﬁdence intervals in Figure 2(a), none of the methods consistently outperforms unitary scalarization.

Original: In fact, owing to our adoption ofexplicit regularization techniques (see § 5.1) its average performance is superior to that reported in theliterature [41, 53].
Revised: In fact, owing to our adoption of explicit regularization techniques (see § 5.1) its average performance is superior to that reported in the literature [42, 54].

Original: As with Multi-MNIST, Figure 8(a) demonstrates that the cross-entropy loss ofeach task can be driven near to its global optimum by most optimizers.
Revised: As with Multi-MNIST, Figure 9(a) demonstrates that the cross-entropy loss of each task can be driven near to its global optimum by most optimizers.

Original: 4.1.3 Cityscapes
Revised: 4.1.3 Cityscapes

Original: In order to complement the multi-task classiﬁcation experiments for Multi-MNIST and CelebA,we present results for Cityscapes [13], a dataset for semantic understanding of urban street scenes.
Revised: In order to complement the multi-task classiﬁcation experiments for Multi-MNIST and CelebA, we present results for Cityscapes [13], a dataset for semantic understanding of urban street scenes.

Original: We rely on a common encoder architecture from the literature [39, 41] (see appendix C.1),  a singledropout layeris contained in the task-speciﬁc heads [39].
Revised: We rely on a common encoder architecture from the literature [40, 42] (see appendix C.1), with a single dropout layer in the task-speciﬁc heads [40].

Original: As for CelebA, unitary scalarization, IMTL,and PCGrad beneﬁt from more regularization than the other optimizers: we employ λ = 10 − 5 forthese three algorithms, as it resulted in better validation performance on the majority of metrics,and λ = 0 for the remaining methods.
Revised: As for CelebA, unitary scalarization, IMTL, and PCGrad beneﬁt from more regularization than the other optimizers: we employ λ = 10 − 5 for these three algorithms, as it resulted in better validation performance on the majority of metrics, and λ = 0 for the remaining methods.

Original: Figure 3 shows test results for two metrics per task, andthe distribution of the training time per epoch.
Revised: Cityscapes is a heterogeneous MTL problem: it contains tasks of different types whose validation metrics cannot be averaged to perform model selection.

Original: The test model for each run and each metric wasselected as the one associated to the best (maximal or minimal, depending on the metric) validationresult across epochs.
Revised: Considering the lack of an established procedure in this context, we potentially evaluate a different model for each metric, chosen as the one with the best (maximal or minimal, depending on the metric) validation performance across epochs (we perform per-run early stopping).

Original: We therefore perform per-run early stopping for most methods (see Figure 17).
Revised: This procedure maximizes per-task performance, at the cost of increased inference time. If inference time is a priority, an alternative model selection procedure could rely on relative task improvement [28, 41, 48], assuming that per-metric improvements are to be weighted linearly. Nevertheless, any consistently applied model selection scheme serves the main goal of our work: evaluating all SMTOs on a fair ground. Figureshows test results for two metrics per task, and the distribution of the training time per epoch.

Original: As with Multi-MNIST and CelebA, no training algorithm clearly outperforms unitary scalarization(signiﬁcant overlaps across conﬁdence intervals exist), which is again the least expensive method.
Revised: As with Multi-MNIST and CelebA, no training algorithm clearly outperforms unitary scalarization (signiﬁcant overlaps across conﬁdence intervals exist), which is again the least expensive method.

Original: Incontrast with a popular hypothesis [10, 29, 41], this holds in spite of relatively large loss imbalances.
Revised: In contrast with a popular hypothesis [10, 30, 42], this holds in spite of relatively large loss imbalances.

Original: In fact, the loss for the depth task is roughly 10 times smaller than that of the segmentation task:see ﬁgures 17(f)-17(g).
Revised: In fact, the loss for the depth task is roughly 10 times smaller than that of the segmentation task: see ﬁgures 17(f)-17(g).

Original: Nevertheless, both losses are rapidly driven towards their respective globalminima.
Revised: Nevertheless, both losses are rapidly driven towards their respective global [Figure]

Revised: A v e r a g

Original: Unlike CelebA (see Figure 2(b)), IMTL, MGDA and GradDrop are signiﬁcantly slower thanunitary scalarization (factors from 1 . 6 to 2 . 3 ), due to the relatively (compared to the parameter space)large size of z in the employed architecture.
Revised: e s u cc e ss r a t e (a) [Figure] repetitions) [Figure]

Revised: A v e r a g

Revised: e s u cc e ss r a t e

Revised: [Figure]

Revised: [Figure]

Revised: minima.

Revised: Unlike CelebA (see Figure 2(b)), IMTL, MGDA and GradDrop are signiﬁcantly slower than unitary scalarization (factors from 1 . 6 to 2 . 3 ), due to the relatively (compared to the parameter space) large size of z in the employed architecture.

Original: PCGrad, instead, appears to be less expensive ( 30%more than the baseline), demonstrating the beneﬁts of working on ∇ θ L i on this model.
Revised: PCGrad, instead, appears to be less expensive ( 30% more than the baseline), demonstrating the beneﬁts of working on ∇ θ L i on this model.

Original: 4.2 Reinforcement Learning
Revised: 4.2 Reinforcement Learning

Original: For RL experiments, we use Meta-World [63] and the Soft Actor-Critic [20] implementation from [54].
Revised: For RL experiments, we use Meta-World [65] and the Soft Actor-Critic [20] implementation from [55].

Original: Unlike § 4.1, the employed network architecture (see appendix C.1) is fully shared across tasks.
Revised: Unlike § 4.1, the employed network architecture (see appendix C.1) is fully shared across tasks.

Original: Therefore, all SMTO implementations for these experiments rely on per-task gradients with respectto network parameters ∇ θ L i (see § 5).
Revised: Therefore, all SMTO implementations for these experiments rely on per-task gradients with respect to network parameters ∇ θ L i (see § 5).

Original: Among the SMTOs we consider, PCGrad is the only onedeveloped with the RL setting in mind.
Revised: Among the SMTOs we consider, PCGrad is the only one developed with the RL setting in mind.

Original: For fairness and completeness, we add all the other SMTOsfrom the supervised learning experiments, and are the ﬁrst to test these optimizers in the RL setting.
Revised: For fairness and completeness, we add all the other SMTOs from the supervised learning experiments, and are the ﬁrst to test these optimizers in the RL setting.

Original: To stabilize learning, we increase the replay buffer size, a well known technique in single-task RL,add actor l 2 regularization, and modify the reward normalization employed by Sodhani et al.
Revised: To stabilize learning, we increase the replay buffer size, a well known technique in single-task RL, add actor l 2 regularization, and modify the reward normalization employed by Sodhani et al.

Original: The unitary scalarization performance reported by Yu et al.
Revised: The unitary scalarization performance reported by Yu et al.

Original: [64] is considerably lower than thatof Sodhani et al.
Revised: [66] is considerably lower than that of Sodhani et al.

Original: [54], which we believe is due to the lack of reward normalization in the former.
Revised: [55], which we believe is due to the lack of reward normalization in the former.

Original: Sodhani et al.
Revised: Sodhani et al.

Original: [54] keep a moving average of rewards in the environment, with a hyperparameter [Figure] A v e r a g [Figure]
Revised: [55] keep a moving average of rewards in the environment, with a hyperparameter controlling the speed of the moving average.

Original: u cc e ss r a t e

Original: [Figure]

Original: [Figure]

Original: repetitions)

Original: [Figure]

Original: A v e r a g

Original: e s u cc e ss r a t e

Original: [Figure]

Original: controlling the speed of the moving average.

Original: As we show in Figure 16, the learning algorithm issensitive to that hyperparameter.
Revised: As we show in Figure 16, the learning algorithm is sensitive to that hyperparameter.

Original: Moreover, such normalization might make similar transitions havedrastically different rewards stored in the replay buffer.
Revised: Moreover, such normalization might make similar transitions have drastically different rewards stored in the replay buffer.

Original: To alleviate these issues, we store the rawrewards in the buffer, and normalize only when a mini-batch is sampled.
Revised: To alleviate these issues, we store the raw rewards in the buffer, and normalize only when a mini-batch is sampled.

Original: Figure 4 reports the best average success rate across the updates and the runtime for 10,000 updates.
Revised: Figure 4 reports the best average success rate across the updates and the runtime for 10,000 updates.

Original: In addition to these summary statistics, reported for consistency with § 4.1, the learning curves areshown in appendix E. Our MT10 (10 tasks) results in Figure 4(a) show that by stabilizing the baselineusing standard RL techniques, unitary scalarization performs on par with other SMTOs, mirroringour ﬁndings in § 4.1.
Revised: In addition to these summary statistics, reported for consistency with § 4.1, the learning curves are shown in appendix E. Our MT10 (10 tasks) results in Figure 4(a) show that by stabilizing the baseline using standard RL techniques, unitary scalarization performs on par with other SMTOs, mirroring our ﬁndings in § 4.1.

Original: This is in contrast with the previous literature, which reported that PCGradoutperforms unitary scalarization [54, 64].
Revised: This is in contrast with the previous literature, which reported that PCGrad outperforms unitary scalarization [55, 66].

Original: Figure 4(b) presents results on MT50 (50 tasks): similarlyto MT10, none of the SMTOs signiﬁcantly outperforms unitary scalarization, with PCGrad’s averagebeing slightly above unitary scalarization.
Revised: Figure 4(b) presents results on MT50 (50 tasks): similarly to MT10, none of the SMTOs signiﬁcantly outperforms unitary scalarization, with PCGrad’s average being slightly above unitary scalarization.

Original: We speculate that the stochastic loss rescaling performedby PCGrad (see Proposition 3) reduces the differences in task return scales, and expect that methodslike PopArt [59] would have a similar effect without requiring access to per-task gradients.
Revised: We speculate that the stochastic loss rescaling performed by PCGrad (see Proposition 3) reduces the differences in task return scales, and expect that methods like PopArt [60] would have a similar effect without requiring access to per-task gradients.

Original: While wedid not tune hyperparameters for MT50 (we employed those found for MT10), it would be mucheasier to do that for unitary scalarization due to its lower runtime.
Revised: While we did not tune hyperparameters for MT50 (we employed those found for MT10), it would be much easier to do that for unitary scalarization due to its lower runtime.

Original: In fact, Figure 4(d) shows thata single unitary scalarization run takes roughly 15 hours, whereas PCGrad, MGDA and GradDroprequire more than a week.
Revised: In fact, Figure 4(d) shows that a single unitary scalarization run takes roughly 15 hours, whereas PCGrad, MGDA and GradDrop require more than a week.

Original: Similarly to MT10, actor regularization pushes the average performanceof unitary scalarization higher (see in appendix E.2).
Revised: Similarly to MT10, actor regularization pushes the average performance of unitary scalarization higher (see in appendix E.2).

Original: Overall, as in the supervised learning setting,unitary scalarization performs comparably to SMTOs despite being simpler and less demanding inboth memory and compute.
Revised: Overall, as in the supervised learning setting, unitary scalarization performs comparably to SMTOs despite being simpler and less demanding in both memory and compute.

Original: IMTL was unstable on this RL benchmark and all of the runs crasheddue to numerical overﬂow.
Revised: IMTL was unstable on this RL benchmark and all of the runs crashed due to numerical overﬂow.

Original: We hence omit IMTL results from the main body of the paper and showits results in Figure 13 in appendix E, which also describes a possible explanation.
Revised: We hence omit IMTL results from the main body of the paper and show its results in Figure 13 in appendix E, which also describes a possible explanation.

Original: We hypothesizethat the instability of IMTL is due to lack of bounds on scaling coefﬁcients.
Revised: We hypothesize that the instability of IMTL is due to lack of bounds on scaling coefﬁcients.

Original: See appendix C.2 forhyperparameter settings and ablation studies.
Revised: See appendix C.2 for hyperparameter settings and ablation studies.

Original:  Regularization in Specialized Multi-Task Optimizers
Revised:  Regularization in Specialized Multi-Task Optimizers

Original: The empirical results presented in § 4 motivate the need to carefully analyze existing SMTOs.
Revised: The empirical results presented in § 4 motivate the need to carefully analyze existing SMTOs.

Original: Wemake an initial attempt in this direction by viewing their effects through the lens of regularization.
Revised: We make an initial attempt in this direction by viewing their effects through the lens of regularization.

Original: Let us deﬁne a regularizer as a technique to reduce overﬁtting [15].
Revised: Let us deﬁne a regularizer as a technique to reduce overﬁtting [15].

Original: We ﬁrst show that the SMTOsconsidered in § 4 empirically act as regularizers via an ablation study ( § 5.1).
Revised: We ﬁrst show that the SMTOs considered in § 4 empirically act as regularizers via an ablation study ( § 5.1).

Original: We then take a closerlook at their behavior, presenting technical results that support their alternative interpretation asregularizers ( § 5.2).
Revised: We then take a closer look at their behavior, presenting technical results that support their alternative interpretation as regularizers ( § 5.2).

Revised: Finally, § 5.3 provides empirical backing for some of the technical results.

Original: Unless otherwise stated, we assume that MTL methods apply only to θ (cid:107) and thatstandard gradient-based updates are employed for tasks-speciﬁc parameters θ ⊥ .
Revised: Unless otherwise stated, we assume that MTL methods apply only to θ (cid:107) and that standard gradient-based updates are employed for tasks-speciﬁc parameters θ ⊥ .

Original: We furthermoreadopt the following shorthands: L i ( θ ) for L i ( f ( θ , X, i ) , Y ) , and ∇ θ L i for ∇ θ L i ( f ( θ , X, i ) , Y ) .
Revised: We furthermore adopt the following shorthands: L i ( θ ) for L i ( f ( θ , X, i ) , Y ) , and ∇ θ L i for ∇ θ L i ( f ( θ , X, i ) , Y ) .

Original: 5.1 Ablation Study
Revised: 5.1 Ablation Study

Original: We repeat the experiment from § 4.1.2 and remove explicit regularization: no dropout layers areadded to the encoder-decoder architecture, and λ = 0 for all optimizers.
Revised: We repeat the experiment from § 4.1.2 and remove explicit regularization: no dropout layers are added to the encoder-decoder architecture, and λ = 0 for all optimizers.

Original: In addition, we examine thebehavior of two different (cid:96) 2 -regularized instances of unitary scalarization: λ = 10 − 4 for “Unit.
Revised: In addition, we examine the behavior of two different (cid:96) 2 -regularized instances of unitary scalarization: λ = 10 − 4 for “Unit.

Original: Scal.(cid:96) 2 ”, λ = 2 × 10 − 3 for “Unit.
Revised: Scal. (cid:96) 2 ”, λ = 2 × 10 − 3 for “Unit.

Original: Scal.
Revised: Scal.

Original: (cid:96) 2 + ”.
Revised: (cid:96) 2 + ”.

Original: Figure 5 shows that SMTOs behave similarly to an (cid:96) 2 - penalized unitary scalarization.
Revised: Figure 5 shows that SMTOs behave similarly to an (cid:96) 2 -penalized unitary scalarization.

Original: Importantly, SMTOs delay overﬁtting, requiring less early stoppingcompared to unitary scalarization to obtain comparable performance.
Revised: Importantly, SMTOs delay overﬁtting, requiring less early stopping compared to unitary scalarization to obtain comparable performance.

Original: In other words, early stoppingis sufﬁcient for unitary scalarization to perform on par with SMTOs.
Revised: In other words, early stopping is sufﬁcient for unitary scalarization to perform on par with SMTOs.

Original: Finally, overﬁtting is furtherreduced by “Unit.
Revised: Finally, overﬁtting is further reduced by “Unit.

Original: Scal.
Revised: Scal.

Original: Reg.”, which plots the regularized unitary scalarization from § 4.1.2, withdropout layers and a weight decay of λ = 10 − 3 .
Revised: Reg.”, which plots the regularized unitary scalarization from § 4.1.2, with dropout layers and a weight decay of λ = 10 − 3 .

Original: Further results are presented in appendices D.2, D.3.
Revised: Further results are presented in appendix D.2.

Original: 5.2 Technical Results
Revised: 5.2 Technical Results

Original: [Figure]

Original: postpone the onset of overﬁtting, mirroring the effect of (cid:96) 2 regularization on unitary scalarization.

Original: All the methods considered in § 5.1 regularizemore than unitary scalarization.
Revised: All the methods considered in § 5.1 regularize more than unitary scalarization.

Original: While RLWwas shown to reduce overﬁtting by the originalauthors [39, theorem 2], we now provide acollection of novel and existing technicalresults that potentially explain the regularizingbehavior of each of the other algorithms,complementing the presentation from § 3.
Revised: While RLW was shown to reduce overﬁtting by the original authors [40, theorem 2], we now provide a collection of novel and existing technical results that potentially explain the regularizing behavior of each of the other algorithms, complementing the presentation from § 3.

Original: Inparticular, we show that MGDA, IMTL and
Revised: In particular, we show that MGDA, IMTL and

Original: PCGrad have a larger convergence set thanunitary scalarization, reducing the chances toland on sharp local minima [15].
Revised: PCGrad have a larger convergence set than unitary scalarization, reducing the chances to land on sharp local minima [15].

Original: Furthermore, GradDrop and PCGrad introduce signiﬁcant stochasticity, which is often linked to the sameeffect [30, 33].
Revised: Furthermore, GradDrop and PCGrad introduce signiﬁcant stochasticity, which is often linked to the same effect [31, 34].

Original: We hope these observations will steer further research.
Revised: We hope these observations will steer further research.

Original: MGDA Let us denote the convex hull of a set A by Conv ( A ) .
Revised: MGDA Let us denote the convex hull of a set A by Conv ( A ) .

Original: We now recall a well-known propertyof MGDA [14] and relate it to the behavior of unitary scalarization.
Revised: We now recall a well-known property of MGDA [14] and relate it to the behavior of unitary scalarization.

Original: Proposition 1.
Revised: Proposition 1.

Original: The MGDA SMTO [53] converges to a superset of the convergence points of unitaryscalarization.
Revised: The MGDA SMTO [54] converges to a superset of the convergence points of unitary scalarization.

Original: More speciﬁcally, it converges to any point θ ∗ (cid:107)
Revised: More speciﬁcally, it converges to any point θ ∗ (cid:107)

Original: such that: 0 ∈ Conv ( {∇ θ ∗ (cid:107) L i | i ∈ T } ) .
Revised: such that: 0 ∈ Conv ( {∇ θ [Equation] (cid:107) L i | i ∈ T } ) .

Original: See appendix B.1 for a simple proof.
Revised: See appendix B.1 for a simple proof.

Original: As a consequence of Proposition 1, MGDA does not necessarilyreach a stationary point for L MT (that is, a point for which (cid:80) i ∈T ∇ θ (cid:107) L i = 0 ) or for any of the losses
Revised: As a consequence of Proposition 1, MGDA does not necessarily reach a stationary point for L MT (that is, a point for which (cid:80) i ∈T ∇ θ (cid:107) L i = 0 ) or for any of the losses

Original: L i ( ∇ θ (cid:107) L i = 0 ).
Revised: L i ( ∇ θ (cid:107) L i = 0 ).

Original: For example, any point θ (cid:107) for which two per-task gradients point in oppositedirections is Pareto stationary.
Revised: For example, any point θ (cid:107) for which two per-task gradients point in opposite directions is Pareto stationary.

Original: On account of the well-known [15] relationship between underoptimizing (e.g., early stopping [7, 38]) and overﬁtting, proposition 1 supports the interpretation of
Revised: On account of the well-known [15] relationship between underoptimizing (e.g., early stopping [7, 39]) and overﬁtting, proposition 1 supports the interpretation of

Original: MGDA as a regularizer for equation (1).
Revised: MGDA as a regularizer for equation (1).

Revised: Empirical evidence that MGDA under-optimizes is provided

Revised: [Figure]

Revised: v e r a g e T a s k

Original: Empirical evidence that MGDA under-optimizes is providedin Figure 8(a), appendix D.3, and Figure 5, which shows over-regularization.
Revised: [Figure] [Figure] [Figure] in § 5.3, Figure 9(a), and Figure 5, which shows over-regularization.

Original: Proposition 1 can betrivially extended to the recent Nash-MTL, which shares the same convergence set [47, Theorem 5.4]
Revised: Proposition 1 can be trivially extended to the recent Nash-MTL, which shares the same convergence set [48, Theorem 5.4].

Original: IMTL We now show that aggregating per-task gradients so that their cosine similarity is the same(equation (4)) yields a constrained steepest-descent algorithm (Proposition 2).
Revised: IMTL We now show that aggregating per-task gradients so that their cosine similarity is the same (equation (4)) yields a constrained steepest-descent algorithm (Proposition 2).

Original: This view on theupdate step of IMTL leads to a novel analysis of its convergence points (corollary 1).
Revised: This view on the update step of IMTL leads to a novel analysis of its convergence points (corollary 1).

Original: Proofs can befound in appendix B.2.
Revised: Proofs can be found in appendix B.2.

Original: We will denote by Aff ( A ) the afﬁne hull of a set A .
Revised: We will denote by Aff ( A ) the afﬁne hull of a set A .

Original: Proposition 2.
Revised: Proposition 2.

Original: IMTL by Liu et al.
Revised: IMTL by Liu et al.

Original: [41] updates θ (cid:107) by taking a step in the steepest descent directionwhose cosine similarity with per-task gradients is the same across tasks.
Revised: [42] updates θ (cid:107) by taking a step in the steepest descent direction whose cosine similarity with per-task gradients is the same across tasks.

Original: Corollary 1.
Revised: Corollary 1.

Original: IMTL by Liu et al.
Revised: IMTL by Liu et al.

Original: [41] converges to a superset of the Pareto-optimal points for θ (cid:107)(and hence of the convergence points of the unitary scalarization).
Revised: [42] converges to a superset of the Pareto-stationary points for θ (cid:107) (and hence of the convergence points of the unitary scalarization).

Original: More speciﬁcally, it converges toany point θ ∗ (cid:107) such that: 
Revised: More speciﬁcally, it converges to any point θ ∗ (cid:107) such that: 0 ∈ Aff (cid:16)(cid:110) [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: As seen for MGDA, corollary 1 implies that, even if the employed model f has the capacity toreach the minimal loss on L MT , IMTL may stop before reaching a stationary point.
Revised: As seen for MGDA, corollary 1 implies that, even if the employed model f has the capacity to reach the minimal loss on L MT , IMTL may stop before reaching a stationary point.

Original: Recalling therelationship between under-optimizing and overﬁtting [15], this supports the interpretation of IMTLas a regularizer for equation (1).
Revised: Recalling the relationship between under-optimizing and overﬁtting [15], this supports the interpretation of IMTL as a regularizer for equation (1).

Original: This is empirically shown in  Figures 5, 8(a), 11.
Revised: This is empirically shown in § 5.3, Figures 5, 9(a).

Original: In particular, unitaryscalarization reaches the same average performance of IMTL but requires earlier stopping.
Revised: In particular, unitary scalarization reaches the same average performance of IMTL but requires earlier stopping.

Original: PCGrad
Revised: PCGrad

Original: We provide an alternative characterization of the PCGrad update rule, highlighting itsstochasticity in the context of its interpretation as loss rescaling [39, 41].
Revised: We provide an alternative characterization of the PCGrad update rule, highlighting its stochasticity in the context of its interpretation as loss rescaling [40, 42].

Original: See appendix B.3 for a proof.338 Proposition 3.
Revised: See appendix B.3 for a proof. Proposition 3.

Original: PCGrad is equivalent to a dynamic, and possibly stochastic, loss rescaling for θ (cid:107) .
Revised: PCGrad is equivalent to a dynamic, and possibly stochastic, loss rescaling for θ (cid:107) .

Original: At each iteration, per-task gradients are rescaled as follows:
Revised: At each iteration, per-task gradients are rescaled as follows:

Original: [Equation]
Revised: [Equation]

Original: (cid:21) .

Original: Furthermore, if |T | > 2 , d ji is a random variable, and the above range contains its support.
Revised: Furthermore, if |T | > 2 , d ji is a random variable, and the above range contains its support.

Original: The results from proposition 3 can be easily extended to GradVac [61], which generalizes PCGrad’sprojection onto the normal vector to arbitrary target cosine similarities between per-task gradients.
Revised: The results from proposition 3 can be easily extended to GradVac [62], which generalizes PCGrad’s projection onto the normal vector to arbitrary target cosine similarities between per-task gradients.

Original: When |T | > 2 , PCGrad corresponds to a stochastic loss re-weighting.
Revised: When |T | > 2 , PCGrad corresponds to a stochastic loss re-weighting.

Original: As such, PCGrad bears manysimilarities with Random Loss Weighting (RLW) [39].
Revised: As such, PCGrad bears many similarities with Random Loss Weighting (RLW) [40].

Original: RLW proposes to sample scalarization weightsfrom standard probability distributions at each iteration, and proves that this leads the better generalization [39, theorem 2].
Revised: RLW proposes to sample scalarization weights from standard probability distributions at each iteration, and proves that this leads the better generalization [40, theorem 2].

Original: Indeed, it is well-known that adding noise to stochastic gradient estimationsleads the optimization towards ﬂatter minima, and that such minima may reduce overﬁtting [30, 33].
Revised: Indeed, it is well-known that adding noise to stochastic gradient estimations leads the optimization towards ﬂatter minima, and that such minima may reduce overﬁtting [31, 34].

Original: In line with the main technical results by Yu et al.
Revised: In line with the main technical results by Yu et al.

Original: [64], we now restrict our focus to two-task problems,which allow for an easy description of PCGrad’s convergence points.
Revised: [66], we now restrict our focus to two-task problems, which allow for an easy description of PCGrad’s convergence points.

Original: The result is largely basedon [64, theorem 1]: we relax some of the assumptions and provide a proof in appendix B.3.
Revised: The result is largely based on [66, theorem 1]: we relax some of the assumptions and provide a proof in appendix B.3.

Original: Corollary 2.
Revised: Corollary 2.

Original: If |T | = 2 , PCGrad will stop at any point where cos( ∇ θ (cid:107) L 1 , ∇ θ (cid:107) L 2 ) = − 1 .
Revised: If |T | = 2 , PCGrad will stop at any point where cos( ∇ θ (cid:107) L 1 , ∇ θ (cid:107) L 2 ) = − 1 .

Original: Furthermore, if L 1 and L 2 are differentiable, and ∇ θ (cid:107) L MT is L-Lipschitz with L > 0 ,PCGrad with step sizet <
Revised: Furthermore, if L 1 and L 2 are differentiable, and ∇ θ (cid:107) L MT is L-Lipschitz with L > 0 ,

Original:  1 L converges to a superset of the convergence points of the unitary scalarization.
Revised: PCGrad with step size t < 1 L converges to a superset of the convergence points of the unitary scalarization.

Original: Corollary 2 implies that, when |T | = 2 , PCGrad may under-optimize equation (1) as MGDAand IMTL.
Revised: Corollary 2 implies that, when |T | = 2 , PCGrad may under-optimize equation (1) as MGDA and IMTL.

Original: In particular, if cos( ∇ θ (cid:107) L 1 , ∇ θ (cid:107) L 2 ) = − 1 , then 0 ∈ Conv ( {∇ θ (cid:107) L 1 , ∇ θ (cid:107) L 2 } ) (seeproposition 1).
Revised: In particular, if cos( ∇ θ (cid:107) L 1 , ∇ θ (cid:107) L 2 ) = − 1 , then 0 ∈ Conv ( {∇ θ (cid:107) L 1 , ∇ θ (cid:107) L 2 } ) (see proposition 1).

Original: We believe that PCGrad’s stochasticity and enlarged convergence set potentiallyexplain its regularizing effect.
Revised: We believe that PCGrad’s stochasticity and enlarged convergence set potentially explain its regularizing effect.

Original: GradDrop While the motivation behind GradDrop is to avoid entry-wise gradient conﬂicts acrosstasks, the main property of the method is to drive the optimization towards “joint minima": points thatare stationary for all the individual tasks at once [11, proposition 1].
Revised: GradDrop While the motivation behind GradDrop is to avoid entry-wise gradient conﬂicts across tasks, the main property of the method is to drive the optimization towards “joint minima": points that are stationary for all the individual tasks at once [11, proposition 1].

Original: In other words: ∇ θ (cid:107) L i = 0 ∀ i ∈
Revised: In other words: ∇ θ (cid:107) L i = 0 ∀ i ∈

Original: T .
Revised: T .

Original: While this property is desirable, we show that it holds beyond GradDrop, and independently ofthe gradient directions.
Revised: While this property is desirable, we show that it holds beyond GradDrop, and independently of the gradient directions.

Original: Under strong assumptions on the model capacity, the above property wouldtrivially hold for unitary scalarization (proposition 5, appendix B.3).
Revised: Under strong assumptions on the model capacity, the above property would trivially hold for unitary scalarization (proposition 5, appendix B.4).

Original: Proposition 4 shows that it holdsfor a simple randomized version of unitary scalarization, which we name Random Grad Drop (RGD).
Revised: Proposition 4 shows that it holds for a simple randomized version of unitary scalarization, which we name Random Grad Drop (RGD).

Original: Proposition 4.
Revised: Proposition 4.

Original: Let L RGD ( θ (cid:107) ) :
Revised: Let L RGD ( θ (cid:107) ) :

Original: = (cid:80) i ∈T u i L
Revised: = (cid:80) i ∈T u i L

Original: i ( θ (cid:107) ) , where u i ∼ Bernoulli ( p ) ∀ i ∈ T and p ∈ (0 , 1] .
Revised: i ( θ (cid:107) ) , where u i ∼ Bernoulli ( p ) ∀ i ∈ T and p ∈ (0 , 1] .

Original: The gradient ∇ θ (cid:107) L RGD is always zero if and only if ∇ θ (cid:107) L i =
Revised: The gradient ∇ θ (cid:107) L RGD is always zero if and only if ∇ θ (cid:107) L i =

Original: 0 ∀ i ∈ T .
Revised: 0 ∀ i ∈ T .

Original: In other words, the resultfrom [11, proposition 1] can be obtained without any information on the sign of per-task gradients.
Revised: In other words, the result from [11, proposition 1] can be obtained without any information on the sign of per-task gradients.

Original: Proposition 4 (see appendix B.4 for a simple proof) shows that an inexpensive sign-independentstochastic scalarization shares GradDrop’s main reported property.
Revised: Proposition 4 (see appendix B.4 for a simple proof) shows that an inexpensive sign-independent stochastic scalarization shares GradDrop’s main reported property.

Original: L RGD can be directly castan instance of RLW, and hence as a regularization method [30, 33].
Revised: L RGD can be directly cast an instance of RLW, and hence as a regularization method [31, 34].

Original: Furthermore, Figure 12 inappendix D.4 shows that the empirical results of GradDrop on CelebA [43] are closely matchedby a sign-agnostic gradient masking, partly undermining the conﬂicting gradients assumption.
Revised: Furthermore, Figure 12 in appendix D.3 shows that the empirical results of GradDrop on CelebA [44] are closely matched by a sign-agnostic gradient masking, partly undermining the conﬂicting gradients assumption.

Original: Webelieve that the above results, along with the authors’ original experiments showing that GradDropdelays overﬁtting on CelebA [11, ﬁgure 3], suggest that GradDrop behaves as a regularizer. 
Revised: We believe that the above results, along with the authors’ original experiments showing that GradDrop delays overﬁtting on CelebA [11, ﬁgure 3], suggest that GradDrop behaves as a regularizer. 5.3 Under-Optimization: Empirical Study

Revised: As seen in § 5.2, MGDA and IMTL might under-optimize equation (1) compared to unitary scalarization due to their larger convergence sets.

Original: Conclusions
Revised: In order to assess whether this is empirically the case, we estimate (cid:13)(cid:13)(cid:80) i ∈T ∇ θ (cid:107) L i (cid:13)(cid:13) 2 , the norm of the unitary scalarization update on shared parameters θ (cid:107) , for all optimizers throughout the unregularized CelebA experiment from § 5.1. Large magnitudes for (cid:13)(cid:13)(cid:80) i ∈T ∇ θ (cid:107) L i (cid:13)(cid:13) 2 towards convergence would indicate that SMTOs steer optimization far from stationary points of unitary scalarization, resulting in under-optimization. We compute the update norm on the mini-batch loss every 100 updates, and report the per-epoch average in Figure 6. Most SMTOs have a smaller update magnitude than unitary scalarization in the ﬁrst 15 epochs.

Revised: However, towards convergence, SMTOs display larger (cid:13)(cid:13)(cid:80) i ∈T ∇ θ (cid:107) L i (cid:13) (cid:13) 2 compared to unitary scalarization.

Revised: In particular, IMTL and MGDA have the largest norm, denoting signiﬁcant empirical under-optimization.

Revised: The additional stochasticity of RLW, PCGrad, and GradDrop also appears to lead to larger norm values than unitary scalarization, yet to a lesser degree.

Revised: Given that MGDA and IMTL incur a larger loss than unitary scalarization in later epochs (see Figure 9(a) in appendix D.2), we can conclude that they guide optimization towards regions of the parameter space that under-optimize equation (1), providing empirical support for our analysis.

Revised:  Conclusions

Original: This paper made two main contributions.
Revised: This paper made two main contributions.

Original: First, we evaluated popular SMTOs using a single experimental pipeline, including previously unpublished results of MGDA, IMTL, RLW, and GradDrop inthe RL setting.
Revised: First, we evaluated popular SMTOs using a single experimental pipeline, including previously unpublished results of MGDA, IMTL, RLW, and GradDrop in the RL setting.

Original: Surprisingly, our evaluation showed that none of the SMTOs consistently outperformunitary scalarization, the simplest and least expensive method.
Revised: Surprisingly, our evaluation showed that none of the SMTOs consistently outperform unitary scalarization, the simplest and least expensive method.

Original: Second, in order to explain our surprising results, we postulate that SMTOs act as regularizers and present an analysis that supports ourhypothesis.
Revised: Second, in order to explain our surprising results, we postulate that SMTOs act as regularizers and present an analysis that supports our hypothesis.

Original: We believe our work calls for further reevaluation of progress in developing principledand efﬁcient MTL algorithms.
Revised: We believe our work calls for further reevaluation of progress in developing principled and efﬁcient MTL algorithms.

Original: We conclude by addressing the limitations of our work.
Revised: We conclude by addressing the limitations of our work.

Original: While we covered a wide range of popularbenchmarks, we do not exclude the existence of settings where unitary scalarization underperforms:discovering them is an interesting direction for future work.
Revised: While we covered a wide range of popular benchmarks, we do not exclude the existence of settings where unitary scalarization underperforms: discovering them is an interesting direction for future work.

Original: Furthermore, our experimental resultswere obtained via grid searches under limited compute resources: some of the methods mightbeneﬁt from further ﬁne-tuning.
Revised: Furthermore, our experimental results were obtained via grid searches under limited compute resources: some of the methods might beneﬁt from further ﬁne-tuning.

Original: Nevertheless, we remark that ﬁne-tuning will be easier for unitaryscalarization due to its shorter runtimes.
Revised: Nevertheless, we remark that ﬁne-tuning will be easier for unitary scalarization due to its shorter runtimes.

Original: Finally, we presented the regularization hypothesis only as apartial explanation of our results: we hope it will steer further analysis and consequently improve theunderstanding of MTL.
Revised: Finally, we presented the regularization hypothesis only as a partial explanation of our results: we hope it will steer further analysis and consequently improve the understanding of MTL.

Original: Checklist
Revised: Acknowledgements

Original: For all
Revised: VK was funded by Samsung R&D Institute UK through the EPSRC Centre for Doctoral Training (CDT) in Autonomous Intelligent Machines and Systems (AIMS) at the University of Oxford . ADP was funded by EPSRC for the AIMS CDT, grant EP/L015987/1, and by an IBM PhD fellowship. SW has received funding from the European Research Council under the European Union’s Horizon 2020 research and innovation programme (grant agreement number 637713). The experiments were made possible by a generous equipment grant from NVIDIA.

Original: authors...
Revised: We would like to thank Lin et al. [40], Sodhani et al. [55] and Sener and Koltun [54] for publicly releasing their code. The authors thank Kristian Hartikainen for helpful comments on the RL experiments.

Original:  (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] see § 6.
Revised: [Yes] see § 6.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] due to space constraints, we provide a discussion in appendix A. (d) Have you read the ethics review guidelines and ensured that your paper conforms to 
Revised: [Yes] due to space constraints, we provide a discussion in appendix A. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: them? [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] we provide full proofs in the Appendix, and refer to them in the main body of the paper.
Revised: [Yes] we provide full proofs in the Appendix, and refer to them in the main body of the paper.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] we provide thecode and the instructions in the supplemental material.
Revised: [Yes] we provide the code and the instructions in the supplemental material.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] see appendix C.1.
Revised: [Yes] see appendix C.1.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] appendix C.3 describes licenses of all benchmarks and implementations we used for our work.
Revised: [Yes] appendix C.3 describes licenses of all benchmarks and implementations we used for our work.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] we include the code and the instructions on how to replicate the experiments into the supplemental material.
Revised: [Yes] we include the code and the instructions on how to replicate the experiments into the supplemental material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Societal Impact
Revised: A Societal Impact

Original: Due to the object of its study, our work does not have a direct societal impact.
Revised: Due to the object of its study, our work does not have a direct societal impact.

Original: However, as anymachine learning paper, it can potentially negatively effect the society through automation and lossof jobs.
Revised: However, as any machine learning paper, it can potentially negatively effect the society through automation and loss of jobs.

Original: While it is hard to anticipate any particular risk, as any technology, if not regulated properly,it might lead to growing social and economic inequality.
Revised: While it is hard to anticipate any particular risk, as any technology, if not regulated properly, it might lead to growing social and economic inequality.

Original: On the positive side, our work might have a positive environmental impact since it advocates forsimpler and more economical methods which will reduce energy consumption in data centers.
Revised: On the positive side, our work might have a positive environmental impact since it advocates for simpler and more economical methods which will reduce energy consumption in data centers.

Original: Finally,simpler methods are usually easier to understand, which is beneﬁcial in terms of explainability, animportant factor for real-life applications.
Revised: Finally, simpler methods are usually easier to understand, which is beneﬁcial in terms of explainability, an important factor for real-life applications.

Original: B Supplement to the Overview of Multi-Task Optimizers
Revised: B Supplement to the Overview of Multi-Task Optimizers

Original: This section presents the proofs and the technical results omitted from section 5, along with a description of the use of per-task gradients with respect to the last shared activation for encoder-decoderarchitectures (usually less expensive than per-task gradients with respect to shared parameters).
Revised: This section presents the proofs and the technical results omitted from section 5, along with a description of the use of per-task gradients with respect to the last shared activation for encoder-decoder architectures (usually less expensive than per-task gradients with respect to shared parameters).

Original: B.1 MGDA
Revised: B.1 MGDA

Original: Proposition 1.
Revised: Proposition 1.

Original: The MGDA SMTO [53] converges to a superset of the convergence points of unitaryscalarization.
Revised: The MGDA SMTO [54] converges to a superset of the convergence points of unitary scalarization.

Original: More speciﬁcally, it converges to any point θ ∗ (cid:107)
Revised: More speciﬁcally, it converges to any point θ ∗ (cid:107)

Original: such that: 0 ∈ Conv ( {∇ θ ∗ (cid:107) L i | i ∈ T } ) .
Revised: such that: 0 ∈ Conv ( {∇ θ ∗ (cid:107) L i | i ∈ T } ) .

Original: Proof.
Revised: Proof.

Original: As shown by Désidéri [14], equation (3) is a simplex-constrained norm-minimization problem.
Revised: As shown by Désidéri [14], equation (3) is a simplex-constrained norm-minimization problem.

Original: In other words, the argument of the minimum is the projection of 0 onto the feasible set.
Revised: In other words, the argument of the minimum is the projection of 0 onto the feasible set.

Original: Therefore:[-g = 
-]
Revised: Therefore:[-g = 
-]

Original: [-⇐⇒ 
 ∈ Conv ( {∇ θ (cid:107) L i | i ∈ T } ) .-]
Revised: [-⇐⇒ 
 ∈ Conv ( {∇ θ (cid:107) L i | i ∈ T } ) .-][Equation]

Original: It then sufﬁces to point out that (cid:80) i ∈T ∇ θ (cid:107) Li =
Revised: It then sufﬁces to point out that (cid:80) i ∈T ∇ θ (cid:107) L

Original: [-⇐⇒ (cid:80) i ∈T 1 |T | ∇ θ (cid:107) L i = 
-]
Revised: [-⇐⇒ (cid:80) i ∈T 1 |T | ∇ θ (cid:107) L i = 
-][Equation]

Original: ⇒ 0 ∈
Revised: = [Equation]

Original: Conv ( {∇ θ (cid:107) L i | i ∈ T } ) to conclude the proof.
Revised: Conv ( {∇ θ (cid:107) L i | i ∈ T } ) to conclude the proof.

Original: Due to the cost of computing per-task gradients, Sener and Koltun [53] propose MGDA-UB, whichreplaces the gradients wrt the parameters ∇ θ (cid:107) L i with the gradients wrt the shared activation ∇ z L iin the computation of the coefﬁcients of g = − (cid:80) 
Revised: Due to the cost of computing per-task gradients, Sener and Koltun [54] propose MGDA-UB, which replaces the gradients wrt the parameters ∇ θ (cid:107) L i with the gradients wrt the shared activation ∇ z L i in the computation of the coefﬁcients of g = − (cid:80) i

Original: i α i ∇ θ (cid:107) L i .
Revised: α i ∇ θ (cid:107) L i .

Original: This yields an upper bound on theobjective of equation (3), thus restricting the set of points the algorithm convergences to.
Revised: This yields an upper bound on the objective of equation (3), thus restricting the set of points the algorithm convergences to.

Original: Ratherthan directly relying on ∇ θ (cid:107) L i , g can then be obtained by computing the gradient of (cid:80) i ∈T α i L i viareverse-mode differentiation, hence saving memory and compute.
Revised: Rather than directly relying on ∇ θ (cid:107) L i , g can then be obtained by computing the gradient of (cid:80) i ∈T α i L i via reverse-mode differentiation, hence saving memory and compute.

Original: Corollary 3.
Revised: Corollary 3.

Original: The MGDA-UB SMTO by Sener and Koltun [53] converges to any point such that:0 ∈ Conv ( {∇ z L i | i ∈ T } ) .
Revised: The MGDA-UB SMTO by Sener and Koltun [54] converges to any point such that: 0 ∈ Conv ( {∇ z L i | i ∈ T } ) .

Original: Furthermore, if ∂ z ∂ θ (cid:107) is non-singular, it converges to a superset of theconvergence points of the unitary scalarization.
Revised: Furthermore, if ∂ z ∂ θ (cid:107) is non-singular, it converges to a superset of the convergence points of the unitary scalarization.

Original: Proof.
Revised: Proof.

Original: The ﬁrst part of the proof proceeds as the proof of proposition 1, noting that the MGDA-UBupdate is associated to the following problem:
Revised: The ﬁrst part of the proof proceeds as the proof of proposition 1, noting that the MGDA-UB update is associated to the following problem:

Original: [Equation]
Revised: [Equation]

Original: In order to show that a stationary point of the unitary scalarization satisﬁes 0 ∈ Conv ( {∇ z ∗ L i | i ∈
Revised: In order to show that a stationary point of the unitary scalarization satisﬁes 0 ∈ Conv ( {∇ z ∗ L i | i ∈

Original: T } ) , we will assume ∂ z ∂ θ (cid:107) is non-singular, as done by Sener and Koltun [53, theorem 1].
Revised: T } ) , we will assume ∂ z ∂ θ (cid:107) is non-singular, as done by Sener and Koltun [54, theorem 1].

Original: Then, relying on the chain rule, the result follows from:
Revised: Then, relying on the chain rule, the result follows from:

Original: [Equation]
Revised: [Equation]

Original: B.2 IMTL
Revised: B.2 IMTL

Original: Proposition 2.
Revised: Proposition 2.

Original: IMTL by Liu et al.
Revised: IMTL by Liu et al.

Original: [41] updates θ (cid:107) by taking a step in the steepest descent directionwhose cosine similarity with per-task gradients is the same across tasks.
Revised: [42] updates θ (cid:107) by taking a step in the steepest descent direction whose cosine similarity with per-task gradients is the same across tasks.

Original: Proof.
Revised: Proof.

Original: First, equation (4) solves the linear system in α :=
Revised: First, equation (4) solves the linear system in α :=

Original: [ α 1 , . . .
Revised: [ α 1 , . . .

Original: , α m ] given by:
Revised: , α m ] given by:

Original: [Equation]
Revised: [Equation]

Original: which corresponds to ﬁnding a point of A (cid:48)
Revised: which corresponds to ﬁnding a point of A (cid:48)

Original: = Aff ( (cid:8) 
Revised: = Aff ( (cid:8) ∇ θ

Original: [Equation]
Revised: [Equation]

Original: L i | i ∈ T (cid:9) ) which is orthogonal to A :=
Revised: L i | i ∈ T (cid:9) ) which is orthogonal to A :=

Original: Aff 
Revised: Aff (cid:18)(cid:26) ∇ [Equation] L i

Original: [Equation]
Revised: [Equation]

Original: To see this, it sufﬁces to point out that any point orthogonal to A is alsoorthogonal to the vector subspace spanned by differences of vectors belonging to A .
Revised: To see this, it sufﬁces to point out that any point orthogonal to A is also orthogonal to the vector subspace spanned by differences of vectors belonging to A .

Original: As this subspacehas m − 1 dimensions, any vector orthogonal to
Revised: As this subspace has m − 1 dimensions, any vector orthogonal to

Original: [Equation]
Revised: [Equation]

Original: for each i ∈ T \ { 1 } isorthogonal to the entire subspace.
Revised: for each i ∈ T \ { 1 } is orthogonal to the entire subspace.

Original: Second, consider the problem of ﬁnding a point in A that is orthogonal to the afﬁne subspace  A .
Revised: Second, consider the problem of ﬁnding a point in A that is orthogonal to the linear subspace spanned by differences of vectors in A .

Original: Inother words, we seek the projection of 0 onto A .
Revised: In other words, we seek the projection of 0 onto A .

Original: Recalling the deﬁnition of A , we can write:
Revised: Recalling the deﬁnition of A , we can write:

Original: [Equation]
Revised: [Equation]

Original: The solution of equation (7) is always collinear to the solution of equation (4).
Revised: The solution of equation (7) is always collinear to the solution of equation (4).

Original: In fact, if a vectorg ∈ A (cid:48) is orthogonal to the afﬁne subspace A , then γ g = (cid:18)γ (cid:80) [Equation] ∇ [Equation]
Revised: In fact, if a vector g ∈ A (cid:48) is orthogonal to the afﬁne subspace A (or to the linear subspace spanned by differences of its members), then γ g = (cid:18)

Original: ∇ θ

Original: [Equation]
Revised: [Equation]

Original: isorthogonal to A as well, andγ =
Revised: is orthogonal to A as well, and

Original: [Equation]
Revised: [Equation]

Original: Finally, equation (7) differs from equation (3) in two aspects: α is not constrained to be non-negative(hence the convex hull is replaced by the afﬁne hull), and the task vectors are normalized.
Revised: Finally, equation (7) differs from equation (3) in two aspects: α is not constrained to be non-negative (hence the convex hull is replaced by the afﬁne hull) [Equation] and the task vectors are normalized.

Original: Therefore,equation (7) is the dual of:
Revised: Therefore, equation (7) is the dual of:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The proposition then follows by comparing equation (8) with equation (2), and recalling that IMTL-Lonly adds a scaling factor to the chosen update direction.
Revised: The proposition then follows by comparing equation (8) with equation (2), and recalling that IMTL-L only adds a scaling factor to the chosen update direction.

Original: Corollary 1.
Revised: Corollary 1.

Original: IMTL by Liu et al.
Revised: IMTL by Liu et al.

Original: [41] converges to a superset of the Pareto-optimal points for θ (cid:107)(and hence of the convergence points of the unitary scalarization).
Revised: [42] converges to a superset of the Pareto-stationary points for θ (cid:107) (and hence of the convergence points of the unitary scalarization).

Original: More speciﬁcally, it converges toany point θ ∗ (cid:107) such that: 
Revised: More speciﬁcally, it converges to any point θ ∗ (cid:107) such that: 0 ∈ Aff

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Inspecting equation (8), which yields a collinear point to the IMTL update, reveals that
Revised: Inspecting equation (8), which yields a collinear point to the IMTL update, reveals that

Original: IMTL might converge to non Pareto-stationary points: due to the restrictive equality constraints,the minimizer of equation (8) might be 0 even if a descent direction exists.
Revised: IMTL might converge to non Pareto-stationary points: due to the restrictive equality constraints, the minimizer of equation (8) might be 0 even if a descent direction exists.

Original: Furthermore, its dual,equation (7), implies that:
Revised: Furthermore, its dual, equation (7), implies that:

Original: [Equation]
Revised: [Equation]

Original: which, noting that Conv ( A ) ⊆
Revised: which, noting that Conv ( A ) ⊆

Original: Aff ( A ) for any A , concludes the proof.
Revised: Aff ( A ) for any A , concludes the proof.

Original: Similarly to MGDA-UB, Liu et al.
Revised: Similarly to MGDA-UB, Liu et al.

Original: [41] advocate using ∇ z L i in place of ∇ θ [-(cid:
 )-] L i while solvingequation (4), typically reducing the cost of computing the coefﬁcients of g = − (cid:80)i
Revised: [42] advocate using ∇ z L i in place of ∇ θ [-(cid:
 )-](cid:107) L i while solving equation (4), typically reducing the cost of computing the coefﬁcients of g = − (cid:80)

Original: α i ∇ θ
Revised: i α

Original: [Equation] L [Equation]
Revised: i ∇ θ (cid:107) L i .

Original: Corollary 4.
Revised: Corollary 4.

Original: When employing the approximation of problem (4) that relies on ∇ z L i , IMTL by Liuet al.
Revised: When employing the approximation of problem (4) that relies on ∇ z L i , IMTL by Liu et al.

Original: [41] converges to 0 ∈ Aff (cid:16)(cid:110) ∇ z L i (cid:107)∇ z L i (cid:107) | i ∈ T (cid:111)(cid:17) .
Revised: [42] converges to 0 ∈ Aff (cid:16)(cid:110) ∇ z L i ( cid:107)∇ z L i (cid:107) | i ∈ T (cid:111)(cid:17) .

Original: If ∂ z ∂ θ (cid:107) is non-singular, this is a superset of ofthe convergence points of the unitary scalarization.
Revised: If ∂ z ∂ θ (cid:107) is non-singular, this is a superset of of the convergence points of the unitary scalarization.

Original: Proof.
Revised: Proof.

Original: Following the proof of proposition 2, the following problem yields a collinear point to the∇ z L i -approximate IMTL update:
Revised: Following the proof of proposition 2, the following problem yields a collinear point to the ∇ z L i -approximate IMTL update:

Original: [Equation]
Revised: [Equation]

Original: Therefore:
Revised: Therefore:

Original: [Equation]
Revised: [Equation]

Original: Finally, assuming ∂ z ∂ θ (cid:107) is non-singular, we can replicate the procedure in the proof of corollary 3 toget:
Revised: Finally, assuming ∂ z ∂ θ (cid:107) is non-singular, we can replicate the procedure in the proof of corollary 3 to get:

Original: [Equation]
Revised: [Equation]

Original:  ∇ z L i ( cid:107)∇ z L i (cid:107) =
Revised: which shows that Aff (cid:16)(cid:110) ∇ z L i (cid:107)∇ [Equation]

Original: [-⇒ 
-]
Revised: [-⇒ 
-]|

Original: [-∈ Conv (cid:18)(cid:26) ∇ z L i ( cid:107)∇ z L i (cid:107) | i ∈ T (cid:27)(cid:19) ⇒ 
 ∈ Aff (cid:18)(cid:26) ∇ z L i ( cid:107)∇ z L i (cid:107) | i ∈ T (cid:27)(cid:19) , which shows that Aff (cid:16)(cid:110) ∇ z L i (-]
Revised: [-∈ Conv (cid:18)(cid:26) ∇ z L i ( cid:107)∇ z L i (cid:107) | i ∈ T (cid:27)(cid:19) ⇒ 
 ∈ Aff (cid:18)(cid:26) ∇ z L i ( cid:107)∇ z L i (cid:107) | i ∈ T (cid:27)(cid:19) , which shows that Aff (cid:16)(cid:110) ∇ z L i (-][Equation]

Original: cid:107)∇ z L i (cid:107) | i ∈ T (cid:111)(cid:17) contains the convergence points of the unitary scalarization.
Revised: ∈ T (cid:111)(cid:17) contains the convergence points of the unitary scalarization.

Original: B.3 PCGrad
Revised: B.3 PCGrad

Original: Proposition 3.
Revised: Proposition 3.

Original: PCGrad is equivalent to a dynamic, and possibly stochastic, loss rescaling for θ (cid:107) .
Revised: PCGrad is equivalent to a dynamic, and possibly stochastic, loss rescaling for θ (cid:107) .

Original: At each iteration, per-task gradients are rescaled as follows:
Revised: At each iteration, per-task gradients are rescaled as follows:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if |T | > 2 , d ji is a random variable
Revised: Furthermore, if |T | > 2 , d ji is a random variable

Original: [Equation]
Revised: [Equation]

Original: and the above range contains its support.
Revised: and the above range contains its support.

Original: Proof.
Revised: Proof.

Original: We start by pointing out that:
Revised: We start by pointing out that:

Original: [Equation]
Revised: [Equation]

Original: As g i is obtained by iterative projections of ∇ θ (cid:107) L i onto the normals of ∇ θ (cid:107) L j ∀ j ∈ T \ { i } , and thenorm of a vector can only decrease or remain unvaried after projections, we can write the coefﬁcientof the last g i update as:
Revised: As g i is obtained by iterative projections of ∇ θ (cid:107) L i onto the normals of ∇ θ (cid:107) L j ∀ j ∈ T \ { i } [Equation] and the norm of a vector can only decrease or remain unvaried after projections, we can write the coefﬁcient of each g i update as:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if |T | > 2 the contraction factor
Revised: Furthermore [Equation] if |T | > 2 the contraction factor

Original: [Equation]
Revised: [Equation]

Original: for the norm of g i depends on the ordering ofthe projections, which is stochastic by design [64].
Revised: for the norm of g i depends on the ordering of the projections, which is stochastic by design [66].

Original: Therefore, d ij a random variable whose support iscontained in
Revised: Therefore [Equation] d ij a random variable whose support is contained in

Original: [Equation]
Revised: [Equation]

Original: Finally, exploiting the deﬁnition of d ij , we can re-write equation (5) as:
Revised: Finally, exploiting the deﬁnition of d ij , we can re-write equation (5) as:

Original: [Equation]
Revised: [Equation]

Revised: Introducing (and then removing, using their deﬁnition) dummy variables

Revised: d jj = 1 : − g = (cid:88)

Original: [Equation]
Revised: [Equation]

Original:   , from which the result trivially follows.
Revised: from which the result trivially follows.

Original: Corollary 2.
Revised: Corollary 2.

Original: If |T | = 2 , PCGrad will stop at any point where cos( ∇ θ 
Revised: If |T | = 2 , PCGrad will stop at any point where cos( ∇ θ (cid:107) L 1 , ∇ θ (cid:107) L 2 ) = − 1 . Furthermore, if L 1 and L 2 are differentiable

Original: [Equation]
Revised: [Equation]

Original: L 1 , ∇ θ (cid:107) L 2 ) = − 1 .
Revised: and ∇ θ [Table] is

Original: Furthermore, if L 1 and L 2 are differentiable, and ∇ θ (cid:107) L MT is L-Lipschitz with L > 0 , PCGrad with step sizet <
Revised: [Table]

Original: 1 L converges to a superset of the convergence points of the unitary scalarization.
Revised: to [Table] points [Table]

Original: Proof.
Revised: Proof.

Original: Let us start from the ﬁrst statement, which does not require any assumption on the losslandscape.
Revised: Let us start from the ﬁrst statement, which does not require any assumption on the loss landscape.

Original: From proposition 3, we get:
Revised: From proposition 3, we get:

Original: [Equation]
Revised: [Equation]

Original: which shows that, in case of conﬂicting gradient directions, gradient norms are rebalanced proportionally to the angle between them.
Revised: which shows that, in case of conﬂicting gradient directions [Equation] gradient norms are rebalanced proportionally to the angle between them.

Original: For cos( ∇ θ
Revised: For cos( ∇ θ

Original: [Equation]
Revised: [Equation]

Original: L 1 , ∇ θ [Equation] L 2 ) = − 1 , the above evaluates to:
Revised: L 1 , ∇ θ (cid:107) L 2 ) = − 1 , the above evaluates to:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst part of the result then follows by pointing out that, if cos( ∇ θ (cid:107) L 1 , ∇ θ (cid:107) L 2 ) = − 1 , then∇ θ (cid:107) L 1 = −∇ θ (cid:107) L 2 , and hence g = 0 .
Revised: The ﬁrst part of the result then follows by pointing out that, if cos( ∇ θ (cid:107) L 1 , ∇ θ (cid:107) L 2 ) = − 1 , then ∇ θ (cid:107) L 1 = −∇ θ (cid:107) L 2 , and hence g = 0 .

Original: We remark that a similar proof appears in [64, theorem 1 andproposition 1].
Revised: We remark that a similar proof appears in [66, theorem 1 and proposition 1].

Original: However, our derivation relaxes the author’s assumptions on L MT and is thereforeapplicable to the training of neural networks.
Revised: However, our derivation relaxes the author’s assumptions on L MT and is therefore applicable to the training of neural networks.

Original: Finally, given the assumptions on differentiability and smoothness, we need to prove that PCGrad converges to the stationary points of the unitary scalarization: this directly follows from [64, proposition1].
Revised: Finally, given the assumptions on differentiability and smoothness, we need to prove that PCGrad converges to the stationary points of the unitary scalarization: this directly follows from [66, proposition 1].

Original: B.4 GradDrop
Revised: B.4 GradDrop

Original: Proposition 5.
Revised: Proposition 5.

Original: Let us assume, as often demonstrated in the single-task case [1, 44], that the multi-tasknetwork has the capacity to interpolate the data on all tasks at once: min θ L MT =
Revised: Let us assume, as often demonstrated in the single-task case [1, 45], that the multi-task network has the capacity to interpolate the data on all tasks at once: min θ L MT =

Original: (cid:80) i ∈T min θ L i ,and that its training by gradient descent attains such global minimum.
Revised: (cid:80) i ∈T min θ L i , and that its training by gradient descent attains such global minimum.

Original: Then, if inf θ L i > −∞ ∀ i ∈
Revised: Then, if inf θ L i > −∞ ∀ i ∈

Original: T , unitary scalarization converges to a joint minimum.
Revised: T , unitary scalarization converges to a joint minimum.

Original: Proof.
Revised: Proof.

Original: It sufﬁces to point out that if L MT ( θ ∗ ) =
Revised: It sufﬁces to point out that if L MT ( θ ∗ ) =

Original: (cid:80) i ∈T min θ L i , then the globally optimal loss isattained for all tasks.
Revised: (cid:80) i ∈T min θ L i , then the globally optimal loss is attained for all tasks.

Original: In other words L i ( θ ∗ ) =
Revised: In other words L i ( θ ∗ ) =

Original: min
Revised: min

Original: θ L i ∀ i ∈ T , and hence ∇ θ ∗ L i = 0 ∀ i ∈ T(joint minimum).
Revised: θ L i ∀ i ∈ T , and hence ∇ θ ∗ L i = 0 ∀ i ∈ T (joint minimum).

Original: Furthermore, running gradient descent on min θ L MT corresponds to the unitaryscalarization ( § 3), which concludes the proof.
Revised: Furthermore, running gradient descent on min θ L MT corresponds to the unitary scalarization ( § 3), which concludes the proof.

Original: Proposition 4.
Revised: Proposition 4.

Original: Let L RGD ( θ (cid:107) ) :
Revised: Let L RGD ( θ (cid:107) ) :

Original: = (cid:80) i ∈T u i L
Revised: = (cid:80) i ∈T u i L

Original: i ( θ (cid:107) ) , where u i ∼ Bernoulli ( p ) ∀ i ∈ T and p ∈ (0 , 1] .
Revised: i ( θ (cid:107) ) , where u i ∼ Bernoulli ( p ) ∀ i ∈ T and p ∈ (0 , 1] .

Original: The gradient ∇ θ (cid:107) L RGD is always zero if and only if ∇ θ (cid:107) L i =
Revised: The gradient ∇ θ (cid:107) L RGD is always zero if and only if ∇ θ (cid:107) L i =

Original: 0 ∀ i ∈ T .
Revised: 0 ∀ i ∈ T .

Original: In other words, the resultfrom [11, proposition 1] can be obtained without any information on the sign of per-task gradients.
Revised: In other words, the result from [11, proposition 1] can be obtained without any information on the sign of per-task gradients.

Original: Proposition 4 can be proved by adapting the proof from Chen et al.
Revised: Proposition 4 can be proved by adapting the proof from Chen et al.

Original: [11, proposition 1]: it sufﬁces toreplace f ( P ) with the Bernoulli parameter p , which is non-negative by deﬁnition.
Revised: [11, proposition 1]: it sufﬁces to replace f ( P ) with the Bernoulli parameter p , which is non-negative by deﬁnition.

Original: In our opinion, thisseriously undermines the conﬂicting gradient hypothesis that motivated GradDrop.
Revised: In our opinion, this seriously undermines the conﬂicting gradient hypothesis that motivated GradDrop.

Original: For the reader’sconvenience, we now provide a straightforward and self-contained proof.
Revised: For the reader’s convenience, we now provide a straightforward and self-contained proof.

Original: Proof.
Revised: Proof.

Original: Let us start from the statement on ∇ θ (cid:107) L RGD .
Revised: Let us start from the statement on ∇ θ (cid:107) L RGD .

Original: If ∇ θ (cid:107) L i = 0 ∀ i ∈ T , then ∇ θ (cid:107) L RGD =
Revised: If ∇ θ (cid:107) L i = 0 ∀ i ∈ T , then ∇ θ (cid:107) L RGD =

Original: 0with probability one.
Revised: with probability one.

Original: On the other hand, if ∃ j : ∇ [Equation] L j (cid:54) = 0 , then:
Revised: On the other hand, if ∃ j : ∇ θ (cid:107) L j (cid:54) = 0 , then:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality comes from the fact that ∇ θ (cid:107) L RGD = ∇ θ (cid:107) L j is only one of the manyinstances of a non-null ∇ θ (cid:107) L RGD .
Revised: where the ﬁrst inequality comes from the fact that ∇ θ (cid:107) L RGD = ∇ θ (cid:107) L j is only one of the many instances of a non-null ∇ θ (cid:107) L RGD .

Original: Let sign ( x ) stand for the element-wise sign operator applied on x .
Revised: Let sign ( x ) stand for the element-wise sign operator applied on x .

Original: On encoder-decoder architectures,similarly to MGDA and IMTL (see appendices B.1 and B.2), the authors do not apply GradDropon ∇ θ (cid:107) L i , but rather on a the usually less expensive ∇ z L i .
Revised: On encoder-decoder architectures, similarly to MGDA and IMTL (see appendices B.1 and B.2), the authors do not apply GradDrop on ∇ θ (cid:107) L i , but rather on a the usually less expensive ∇ z L i .

Original: In more detail, they compute the
Revised: In more detail, they compute the

Original: GradDrop sign purity scores p from equation (6) on (cid:80) ni
Revised: GradDrop sign purity scores p from equation (6) on (cid:80) ni

Original: =1 ( sign ( z ) (cid:12) ∇ z L i )
Revised: =1 ( sign ( z ) (cid:12) ∇ z L i )

Original: [ i ] ∈ R r , and then applyequation (6) on the ∇ z L i gradients, yielding a vector g z ∈ R n × r .
Revised: [ i ] ∈ R r , and then apply equation (6) on the ∇ z L i gradients, yielding a vector g z ∈ R n × r .

Original: Then, relying on reverse-modedifferentiation, the update direction in the space of the parameters θ (cid:107) is obtained via a Jacobian-vectorproduct: g = − (cid:16) ∂ z ∂ θ (cid:107) (cid:17) T g z .
Revised: Then, relying on reverse-mode differentiation, the update direction in the space of the parameters θ (cid:107) is obtained via a Jacobian-vector product: g = − (cid:16) ∂ z ∂ θ (cid:107) (cid:17) T g z .

Original: Such a computation replaces the similar ∇ θ (cid:107) L MT = (cid:16) ∂ z ∂ θ (cid:107) (cid:17) T ∇ z L MTfrom the unitary scalarization.
Revised: Such a computation replaces the similar ∇ θ (cid:107) L MT = (cid:16) ∂ z ∂ θ (cid:107) (cid:17) T ∇ z L MT from the unitary scalarization.

Original: C Experimental Setting, Reproducibility
Revised: C Experimental Setting, Reproducibility

Original: We now present details concerning the experimental settings from § 4, including details on theemployed open-source software, dataset information, hardware speciﬁcations, and hyper-parameters.
Revised: We now present details concerning the experimental settings from § 4, including details on the employed open-source software, dataset information, hardware speciﬁcations, and hyper-parameters.

Original: C.1 Supervised Learning
Revised: C.1 Supervised Learning

Original: All the experiments were run under Ubuntu 18.04 LTS, on a single GPU per run (using two 8-GPUmachines in total).
Revised: All the experiments were run under Ubuntu 18.04 LTS, on a single GPU per run (using two 8-GPU machines in total).

Original: Timing experiments were all run on Nvidia GeForce GTX 1080 Ti GPUs, with an
Revised: Timing experiments were all run on Nvidia GeForce GTX 1080 Ti GPUs, with an

Original: Intel Xeon E5-2650 CPU.
Revised: Intel Xeon E5-2650 CPU.

Original: The remaining experiments were run on either Nvidia GeForce RTX[-
-]
Revised: The remaining experiments were run on either Nvidia GeForce RTX[-
-]

Original: Ti GPUs or Nvidia GeForce GTX 1080 Ti GPUs, respectively using an Intel Xeon Gold 6230 CPUor an Intel Xeon E5-2650 CPU.
Revised: Ti GPUs or Nvidia GeForce GTX 1080 Ti GPUs, respectively using an Intel Xeon Gold 6230 CPU or an Intel Xeon E5-2650 CPU.

Original: C.1.1 MultiMNIST
Revised: C.1.1 MultiMNIST

Original: Multi-MNIST, originally introduced by Sabour et al.
Revised: Multi-MNIST, originally introduced by Sabour et al.

Original: [51] and as modiﬁed by Sener and Koltun[53], is a simple two-task supervised learning benchmark dataset constructed by uniformly sampling
Revised: [52] and as modiﬁed by Sener and Koltun [54], is a simple two-task supervised learning benchmark dataset constructed by uniformly sampling

Original: MNIST [37] images, and placing one in the top-left corner, the other in the bottom-right corner.
Revised: MNIST [38] images, and placing one in the top-left corner, the other in the bottom-right corner.

Original: Eachof the two overlaid images corresponds to a 10-class classiﬁcation task.
Revised: Each of the two overlaid images corresponds to a 10-class classiﬁcation task.

Original: Using the above procedure,we generate the Multi-MNIST training set from the ﬁrst 50000 MNIST training images, the validationset from the last 10000 training images, and the test set from the original MNIST test set.
Revised: Using the above procedure, we generate the Multi-MNIST training set from the ﬁrst 50000 MNIST training images, the validation set from the last 10000 training images, and the test set from the original MNIST test set.

Original: Forconsistency with the experimental setup of Sener and Koltun [53], we employ a modiﬁed encoderdecoder version of the LeNet architecture [37].
Revised: For consistency with the experimental setup of Sener and Koltun [54], we employ a modiﬁed encoderdecoder version of the LeNet architecture [38].

Original: Speciﬁcally, the last layer is omitted from the encoder,and two fully-connected layers are employed as task-speciﬁc predictive heads.
Revised: Speciﬁcally, the last layer is omitted from the encoder, and two fully-connected layers are employed as task-speciﬁc predictive heads.

Original: The cross-entropyloss is used for both tasks.
Revised: The cross-entropy loss is used for both tasks.

Original: All methods are trained for 100 epochs using Adam [32] in the stochasticgradient setting, with an initial learning rate of η = 10 − 2 (tuned in η ∈ { 10 − 3 , 10 − 2 , 10 − 1 } andyielding the best validation results for all considered algorithms), exponentially decayed by 0 .
Revised: All methods are trained for 100 epochs using Adam [33] in the stochastic gradient setting, with an initial learning rate of η = 10 − 2 (tuned in η ∈ { 10 − 3 , 10 − 2 , 10 − 1 } and yielding the best validation results for all considered algorithms), exponentially decayed by 0 .

Original: 95 aftereach epoch, and a mini-batch size of 256 .
Revised: 95 after each epoch, and a mini-batch size of 256 .

Original: C.1.2 CelebA
Revised: C.1.2 CelebA

Original: The CelebA [43] dataset consists of 200 , 000 headshots (with standard training, validation and testsplits) associated with the presence or absence of 40 attributes.
Revised: The CelebA [44] dataset consists of 200 , 000 headshots (with standard training, validation and test splits) associated with the presence or absence of 40 attributes.

Original: In the MTL literature, is commonlytreated as a 40 -task classiﬁcation problem, each task being a binary classiﬁcation problem foran attribute.
Revised: In the MTL literature, is commonly treated as a 40 -task classiﬁcation problem, each task being a binary classiﬁcation problem for an attribute.

Original: As commonly done in previous work [41, 53, 64], we employ an encoder-decoderarchitecture where the encoder is a ResNet-18 [21] (without the ﬁnal layer), and the per-task decodersare linear classiﬁers.
Revised: As commonly done in previous work [42, 54, 66], we employ an encoder-decoder architecture where the encoder is a ResNet-18 [21] (without the ﬁnal layer) with batch normalization layers [26], and the per-task decoders are linear classiﬁers.

Original: The cross-entropy loss is used for all tasks.
Revised: The cross-entropy loss is used for all tasks.

Original: The training is performed fromscratch for 50 epochs using Adam, with a mini-batch size of 128 and a per-epoch exponential decayfactor of 0 .
Revised: The training is performed from scratch for 50 epochs using Adam, with a mini-batch size of 128 and a per-epoch exponential decay factor of 0 .

Original: As common on this network-dataset combination [11, 39], the initial learning rate isη = 10 − 3 for all methods except for MGDA and IMTL, for which η = 5 × 10 − 4 yielded a bettervalidation performance.
Revised: As common on this network-dataset combination [11, 40], the initial learning rate is η = 10 − 3 for all methods except for MGDA and IMTL, for which η = 5 × 10 − 4 yielded a better validation performance.

Original: As done by the respective authors, for PCGrad, RLW and GradDrop we usethe same learning rate as the unitary scalarization [11, 39, 64].
Revised: As done by the respective authors, for PCGrad, RLW and GradDrop we use the same learning rate as the unitary scalarization [11, 40, 66].

Original: C.1.3 Cityscapes
Revised: C.1.3 Cityscapes

Original: We rely on the version of the dataset pre-processed by Liu et al.
Revised: We rely on the version of the dataset pre-processed by Liu et al.

Original: [42], which consists of 2 , 975training and 500 test images and presents two tasks: semantic segmentation on 7 classes, and depthestimation.
Revised: [43], which consists of 2 ,training and 500 test images and presents two tasks: semantic segmentation on 7 classes, and depth estimation.

Original: We further split the original training set into a validation set of 595 images, employedto tune hyper-parameters, and a training set of 2380 images.
Revised: We further split the original training set into a validation set of 595 images, employed to tune hyper-parameters, and a training set of 2380 images.

Original: Consistently with recent work [39],we rely on a dilated ResNet-50 architecture pre-trained on ImageNet [62] for the encoder, and onthe Atrous Spatial Pyramid Pooling [8] as decoders.
Revised: Consistently with recent work [40], we rely on a dilated ResNet-50 architecture pre-trained on ImageNet [64] for the encoder, and on the Atrous Spatial Pyramid Pooling [8], which internally uses batch normalization, as decoders.

Original: While more powerful encoders might lead tobetter performance on Cityscapes, like the SegNet [2] used in [27, 40, 47], we aim to provide a faircomparison of MTL optimizers, rather than maximize overall task performance.
Revised: While more powerful encoders might lead to better performance on Cityscapes, like the SegNet [2] used in [28, 41, 48], we aim to provide a fair comparison of MTL optimizers, rather than maximize overall task performance.

Original: Cross-entropy lossis employed on the semantic segmentation task, whereas the (cid:96) 1 loss is used for the depth estimation.
Revised: Cross-entropy loss is employed on the semantic segmentation task, whereas the (cid:96)loss is used for the depth estimation.

Original: The training is performed by using Adam with a mini-batch size of 32 for 100 epochs, with an initialstep size η = 5 × 10 − 4 resulting in the best validation performance for all algorithms, exponentiallydecayed by 0 .
Revised: The training is performed by using Adam with a mini-batch size of 32 for 100 epochs, with an initial step size η = 5 × 10 − 4 resulting in the best validation performance for all algorithms, exponentially decayed by 0 .

Original: 95 at each epoch.
Revised: 95 at each epoch.

Original: Cityscapes is a heterogeneous MTL problem: it contains tasks of different types whose validationmetrics cannot be averaged to perform model selection.

Original: Considering the lack of an established model selection procedure in this context, we potentially evaluate a different model for each metric,chosen as the one with the best validation performance across the epochs.

Original: This procedure maximizesper-task performance, at the cost of increased inference time.

Original: If inference time is a priority, analternative model selection procedure could rely on relative task improvement [27, 40, 47], assumingthat per-metric improvements are to be weighted linearly.

Original: Nevertheless, any consistently appliedmodel selection scheme serves the main goal of our work: evaluating all SMTOs on a fair ground.

Original: C.2 Reinforcement Learning
Revised: C.2 Reinforcement Learning

Original: Similarly to the supervised learning experiments, we ran all the experiments under Ubuntu 18.04 LTSusing one GPU per run (using six 8-GPU machines in total).
Revised: Similarly to the supervised learning experiments, we ran all the experiments under Ubuntu 18.04 LTS using one GPU per run (using six 8-GPU machines in total).

Original: Timing experiments were all run using
Revised: Timing experiments were all run using

Original: NVIDIA GeForce RTX 2080 Ti GPUs, with an Intel Xeon Gold 6230 CPU.
Revised: NVIDIA GeForce RTX 2080 Ti GPUs, with an Intel Xeon Gold 6230 CPU.

Original: The main bulk of theremaining experiments was run on Nvidia GeForce RTX 2080 Ti GPUs with either Intel Xeon Gold6230 or Intel Xeon Silver 4216.
Revised: The main bulk of the remaining experiments was run on Nvidia GeForce RTX 2080 Ti GPUs with either Intel Xeon Gold 6230 or Intel Xeon Silver 4216.

Original: We utilised NVIDIA GeForce RTX 3080 GPUs with Intel Xeon
Revised: We utilised NVIDIA GeForce RTX 3080 GPUs with Intel Xeon

Original: Gold 6230 CPUs for a small fraction of experiments.
Revised: Gold 6230 CPUs for a small fraction of experiments.

Original: We use Meta-World’s MT10/MT50 for our experiment.
Revised: We use Meta-World’s MT10/MT50 for our experiment.

Original: The benchmark consists of ten/ﬁfty tasks inwhich a simulated robot manipulator has to perform various actions, e.g., pressing a button, openinga door, or pushing the block.
Revised: The benchmark consists of ten/ﬁfty tasks in which a simulated robot manipulator has to perform various actions, e.g., pressing a button, opening a door, or pushing the block.

Original: We use Sodhani et al.
Revised: We use Sodhani et al.

Original: [54] for most of the hyperparameters and listthem in Table 1.
Revised: [55] for most of the hyperparameters and list them in Table 1.

Original: We use bold font where we use a hyperparameter different from Sodhani et al.
Revised: We use bold font where we use a hyperparameter different from Sodhani et al.

Original: Similarly to Sodhani et al.
Revised: Similarly to Sodhani et al.

Original: [54], we use the V 1 version of Metaworld for our experiments 1 .
Revised: [55], we use the V 1 version of Metaworld for our experiments 2 .

Original: Sodhaniet al.
Revised: Sodhani et al.

Original: [54] use a shared entropy loss weight α for PCGrad and separate α for unitary scalarization 2 .
Revised: [55] use a shared entropy loss weight α for PCGrad and separate α for unitary scalarization 3 .

Original: Inour experiments, use shared α for all of the methods for fairness.
Revised: In our experiments, use shared α for all of the methods for fairness.

Original: Since it is a single number (ratherthan a vector), we used unitary scalarization to update α for all SMTOs apart from PCGrad whichwas already implemented in [54].
Revised: Since it is a single number (rather than a vector), we used unitary scalarization to update α for all SMTOs apart from PCGrad which was already implemented in [55].

Original: We use the same network architecture as in Sodhani et al.
Revised: We use the same network architecture as in Sodhani et al.

Original: [54], i.e. a three-layered feedforwardfully-connected network with 400 hidden units per layer for both, the actor and the critic.
Revised: [55], i.e. a three-layered feedforward fully-connected network with 400 hidden units per layer for both, the actor and the critic.

Original: The actoris shared across all tasks as well as the critic.
Revised: The actor is shared across all tasks as well as the critic.

Original: To normalize rewards, we keep track of ﬁrst and second moments in the buffer and normalise therewards by their standard deviation: r (cid:48) i
Revised: To normalize rewards, we keep track of ﬁrst and second moments in the buffer and normalise the rewards by their standard deviation: r (cid:48) i

Original: =
Revised: =

Original: r i / ˆ σ i , where ˆ σ i is the sample standard deviation of therewards for environment i .
Revised: r i / ˆ σ i , where ˆ σ i is the sample standard deviation of the rewards for environment i .

Original: Sodhani et al.
Revised: Sodhani et al.

Original: [54] average the gradient for unitary scalarisation and pcgrad, whereas our SMTOimplementations sum the gradients, i.e. effectively using larger learning rates (apart from MGDA thatassures that all the aggregation weights sum to 1).
Revised: [55] average the gradient for unitary scalarisation and pcgrad, whereas our SMTO implementations sum the gradients, i.e. effectively using larger learning rates (apart from MGDA that assures that all the aggregation weights sum to 1).

Original: We tried reducing the learning rate for SMTOsthat sum (RLW Norm., RLW Diri., and GradDrop) both for MT10 and MT50, but it worked worsefor these methods and we kept the default learning rate for them as well.
Revised: We tried reducing the learning rate for SMTOs that sum (RLW Norm., RLW Diri., and GradDrop) both for MT10 and MT50, but it worked worse for these methods and we kept the default learning rate for them as well.

Original: We had to use a smallerlearning rate for IMTL, because with the default one it crashed at the beginning of training due tonumerical overﬂow.
Revised: We had to use a smaller learning rate for IMTL, because with the default one it crashed at the beginning of training due to numerical overﬂow.

Original: Smaller learning rate did not prevent it from crashing, but this happened muchlater.
Revised: Smaller learning rate did not prevent it from crashing, but this happened much later.

Original: We tried 10 6 , 2 × 10 6 , and 4 × 10 6 for the replay buffer size with the last being superior in termsof stability.
Revised: We tried 10 6 , 2 × 10 6 , and 4 × 10 6 for the replay buffer size with the last being superior in terms of stability.

Original: Additionally, for l 2 actor regularization, we tried 0 .
Revised: Additionally, for l 2 actor regularization, we tried 0 .

Original: 0001 and 0 .
Revised: 0001 and 0 .

Original: 0003 with the latterbeing slightly superior for the baseline.
Revised: 0003 with the latter being slightly superior for the baseline.

Original: We tried the same options for other SMTOs, and picked thebest option for each of the method.
Revised: We tried the same options for other SMTOs, and picked the best option for each of the method.

Original: For MGDA, no regularisation works best, most likely due to astrong regularization effect of the method itself, which is mirrored by our supervised learning results.
Revised: For MGDA, no regularisation works best, most likely due to a strong regularization effect of the method itself, which is mirrored by our supervised learning results.

Original: PCGrad and Graddrop work best with the regularization coefﬁcient of 0 . 0001 .
Revised: PCGrad and Graddrop work best with the regularization coefﬁcient of 0 . 0001 .

Original: Both RLW variantsuse the same coefﬁcient as the baseline ( 0 . 0003 ).
Revised: Both RLW variants use the same coefﬁcient as the baseline ( 0 . 0003 ).

Original: For MT50, we took the best MT10 hyperparameters, and we believe one could obtain even betterresults for unitary scalarisation since it is much faster to tune compared to other SMTOs (e.g. 15hours for unitary scalarisation vs 9 days for PCGrad).
Revised: For MT50, we took the best MT10 hyperparameters, and we believe one could obtain even better results for unitary scalarisation since it is much faster to tune compared to other SMTOs (e.g.hours for unitary scalarisation vs 9 days for PCGrad).

Original: C.3 Software Acknowledgments and Licenses
Revised: C.3 Software Acknowledgments and Licenses

Original: Our codebase is built upon several prior works: [53], [42], [39] and [54]: all of them were releasedunder a MIT license.
Revised: Our codebase is built upon several prior works: [54], [43], [40] and [55]: all of them were released under a MIT license.

Original: We also acknowledge Tseng [58], upon which we built some of our code.
Revised: We also acknowledge Tseng [59], upon which we built some of our code.

Original: Multi-MNIST is based on MNIST dataset that is released under Creative Commons AttributionShare Alike 3.0 license.
Revised: Multi-MNIST is based on MNIST dataset that is released under Creative Commons AttributionShare Alike 3.0 license.

Original: The code for generating Multi-MNIST dataset was taken from Senerand Koltun [53] released under MIT license. CelebA dataset has a custom license allowing noncommercial research purposes. More details can be found on the project website: http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html . Cityscapes also has a custom license allowing noncommercial research purposes.
Revised: The code for generating Multi-MNIST dataset was taken from Sener and Koltun [

Original: The full text of the license can be found on the project website: https://www.cityscapes-dataset.com/license/ .

Original: Metaworld, used for RL experiments is releasedunder MIT license.

Original: D Supplementary Supervised Learning Experiments
Revised: D Supplementary Supervised Learning Experiments

Original: [Table]
Revised: [Table]

Original: [Table]

Original: 2 coeff.

Original: This section presents supervised learning resultsomitted from § 4.1.
Revised: This section presents supervised learning results omitted from § 4.1.

Original: In particular, we show additional plots for the experiments of § 4.1, thenpresent an analysis of the regularising effect of
Revised: In particular, we show additional plots for the experiments of § 4.1, then present an analysis of the regularising effect of

Original: SMTOs in the absence of single-task regularization ( § D.2), and conclude with an ablationstudy on GradDrop’s dependency on the sign ofper-task gradients ( § D.4).
Revised: SMTOs in the absence of single-task regularization ( § D.2) [Table] and conclude with an ablation study on GradDrop’s dependency on the sign of per-task gradients ( § D.3).

Original: D.1 Addendum
Revised: D.1 Addendum

Original: This section complements the plots presentedin § 4.1.
Revised: This section complements the plots presented in § 4.1.

Original: In particular, we show the test andruntime results in table form, along with thebehavior of the validation metrics and of thetraining loss over the training epochs.
Revised: In particular, we show the test and runtime results in table form, along with the behavior of the validation metrics and of the training loss over the training epochs.

Original: Plots for
Revised: Plots for

Original: Multi-MNIST, CelebA, and Cityscapes are reported in Figures 6, 7 and 17, respectively.
Revised: Multi-MNIST, CelebA, and Cityscapes are reported in Figures 7, 8 and 17, respectively.

Original: The
Revised: The

Original: Multi-MNIST plots show that all optimizers arerelatively stable and drive each task’s loss towards its global minimum of 0 .
Revised: Multi-MNIST plots show that all optimizers are relatively stable and drive each task’s loss towards its global minimum of 0 .

Original: The behavior ofthe CelebA training loss demonstrates heavierregularization (compare with the unregularizedplot in Figure 8(a)).
Revised: The behavior of the CelebA training loss demonstrates heavier regularization (compare with the unregularized plot in Figure 9(a)).

Original: Except IMTL and MGDA,for which the tuned values of the weight decayprevent overﬁtting, the other optimizers displayvery similar validation and training curves, andstart overﬁtting around epoch 30 .
Revised: Except IMTL and MGDA, for which the tuned values of the weight decay prevent overﬁtting, the other optimizers display very similar validation and training curves, and start overﬁtting around epoch 30 .

Original: Consideringthat most SMTOs required less regularization(see § 4.1.2), the results are consistent with ourinterpretation of SMTOs as regularizers in § 5.
Revised: Considering that most SMTOs required less regularization (see § 4.1.2), the results are consistent with our interpretation of SMTOs as regularizers in § 5.

Original: The Cityscapes plots display a certain instabilityacross training epochs, as demonstrated by thevarious peaks and valleys in the metrics.
Revised: The Cityscapes plots display a certain instability across training epochs, as demonstrated by the various peaks and valleys in the metrics.

Original: Nevertheless, in spite of a factor 10 difference in scale,both training losses are rapidly driven towards0 by most optimizers.
Revised: Nevertheless, in spite of a factor 10 difference in scale, both training losses are rapidly driven towards 0 by most optimizers.

Original: D.2 Unregularized Experiments
Revised: D.2 Unregularized Experiments

Original: Figures 5, 8(a) and 8(b) respectively report theaverage task validation accuracy, the multi-tasktraining loss, and the multi-task validation lossat each training epoch.
Revised: Figures 5, 9(a) and 9(b) respectively report the average task validation accuracy, the multi-task training loss, and the multi-task validation loss at each training epoch.

Original: The regularizing effectof SMTOs compared to unitary scalarization isshown by: (i) the delay of the onset of overﬁtting on the validation data in ﬁgure 5, (ii) the [Figure] [Figure]
Revised: The regularizing effect of SMTOs compared to unitary scalarization is shown by: (i) the delay of the onset of overﬁtting on the validation data in ﬁgure 5, (ii) the reduction of the convergence rate on the training loss in ﬁgure 9(a) (compare with ﬁgure 8(b)), and (iii) the fact that validation and training losses remain positively correlated for larger numbers of epochs.

Original: [Table]
Revised: In fact, the behavior of both the training and validation loss for the SMTOs closely parallels that of (cid:96) 2 -regularized unitary scalarization, with differing degrees of regularization.

Original: [Figure]
Revised: We further note that unregularized IMTL displays a certain instability (compare with the regularized version in ﬁgure 8(a)).

Original: r
Revised: The addition of dropout layers further reduces overﬁtting, improves stability (reduced conﬁdence intervals) and pushes the average validation curve upwards, motivating its use on all optimizers for the

Original: [Figure]
Revised: [Figure]

Original: g

Original: [Figure]
Revised: [Figure]

Original: T
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: k
Revised: k

Original: [Figure]
Revised: [Figure]

Original: u r
Revised: u r

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: reduction of the convergence rate on the trainingloss in ﬁgure 8(a) (compare with ﬁgure 7(b)),and (iii) the fact that validation and training losses remain positively correlated for larger numbersof epochs. In fact, the behavior of both the training and validation loss for the SMTOs closelyparallels that of (cid:96) 2 -regularized unitary scalarization, with differing degrees of regularization. Wefurther note that unregularized IMTL displays a certain instability (compare with the regularizedversion in ﬁgure 7(a)). The addition of dropout layers further reduces overﬁtting, improves stability (reduced conﬁdenceintervals) and pushes the average validation curve upwards, motivating its use on all optimizers for theexperiments of § 4.1.2.
Revised: experiments of § 4.1.2.

Original: Nevertheless, conﬁdence intervals in Figure 5 still overlap due to the instabilityof the unregularized unitary scalarization.
Revised: Nevertheless, conﬁdence intervals in Figure 5 still overlap due to the instability of the unregularized unitary scalarization.

Original: Figure 10 provides a more detailed comparison over 20repetitions, conﬁrming that the combined use of dropout layers and (cid:96) 2 regularization improves averageperformance and reduces the empirical variance for unitary scalarization.
Revised: Figure 11 provides a more detailed comparison overrepetitions, conﬁrming that the combined use of dropout layers and (cid:96) 2 regularization improves average performance and reduces the empirical variance for unitary scalarization.

Original: Furthermore, Figure 9 shows that regularization improves the peak average validation performance for all algorithms,demonstrating the need of tuning λ also for SMTOs.
Revised: Furthermore, Figureshows that regularization improves the peak average validation performance for all algorithms, demonstrating the need of tuning λ also for SMTOs.

Original: We conclude by pointing out that even withoutregularization, when carefully tuned, the maximal performance over epochs of unitary scalarizationis comparable to SMTOs in Figure 5.[Figure] l t i [Figure]
Revised: We conclude by pointing out that even without regularization, when carefully tuned, the maximal performance over epochs of unitary scalarization is comparable to SMTOs in Figure 5.

Original: a i n i n [Figure] [Figure] [Figure]
Revised: D.3 Sign-Agnostic GradDrop

Original: A v e r a g e T a s k A cc u r a c y D.3 Multi-Task Optimizers and Under-Optimization The experiments in § 5.1 and appendix D.2 show that SMTOs empirically act as regularizers forequation (1).
Revised: We will now present an ablation study on GradDrop, investigating the effect of the sign of per-task gradients on the SMTO’s performance.

Original: The analysis from § 5.2 provides a partial explanation for the results: due to their largerconvergence sets, MGDA, IMTL and PCGrad might under-optimize equation 1 compared to unitaryscalarization. In order to empirically evaluate our analysis, we estimate (cid:13)(cid:13)(cid:80) i ∈T ∇ θ (cid:107) L i (cid:13)(cid:13) 2 , the normof the unitary scalarization update on shared parameters θ (cid:107) , for all optimizers throughout the unregularized CelebA experiment ( § 5.1).
Revised: Speciﬁcally, we compare the performance of GradDrop

Original: Large magnitudes for (cid:13)(cid:13)(cid:80) i ∈T ∇ θ (cid:107) L i (cid:13)(cid:13) 2 towards convergencewould indicate that SMTOs steer optimization far from stationary points of unitary scalarization,resulting in under-optimization. For Figure 11, we compute the update norm on the mini-batch loss every 100 updates, and report theper-epoch average. Most SMTOs have a smaller update magnitude than unitary scalarization in theﬁrst 15 epochs. However, towards convergence, (cid:13)(cid:13)(cid:80) i ∈T ∇ θ (cid:107) L i (cid:13)(cid:13) 2 is larger for SMTOs comparedto unitary scalarization.
Revised: [Figure]

Original: In particular, IMTL and MGDA, both of which were shown to have alarger convergence set in § 5.2, display the largest norm. The additional stochasticity of RLW and GradDrop also appears to lead to larger norm values than unitary scalarization, yet to a lesser degree. Given that SMTOs incur a larger loss than unitary scalarization (see Figure 8(a)) in the later epochs,we can conclude that SMTOs guide optimization towards regions of the parameter space whichunder-optimize the loss, providing empirical support for our analysis.
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: We will now present an ablation study on GradDrop, investigating theeffect of the sign of per-taskgradients on the SMTO’s performance.
Revised: the

Original: Speciﬁcally, we compare the performance of GradDropwith a sign-agnostic version of  its stochastic gradient masking (which we refer to as “Sign-Agnostic
Revised: CelebA dataset: violin plots (20 runs) for the best avg. task validation accuracy over epochs. The width at a given value represents the proportion of runs yielding that result. Regularization improves the average performance while decreasing its variability.

Revised: with a sign-agnostic version of its stochastic gradient masking (which we refer to as “Sign-Agnostic

Original: GradDrop"), whose update direction is deﬁned as follows:
Revised: GradDrop"), whose update direction is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where u i , ∇ z L i ∈ R n × r and, for all i ∈ T , u i is i.i.d.
Revised: where u i , ∇ z L i ∈ R n × r and, for all i ∈ T , u i is i.i.d.

Original: according to u i [ j, k ] ∼ Bernoulli ( p ) ∀ j ∈{ 1 , . . . , n } , k ∈ { 1 , . . .
Revised: according to u i [ j, k ] ∼ Bernoulli ( p ) ∀ j ∈ { 1 , . . . , n } , k ∈ { 1 , . . .

Original: , r } .
Revised: , r } .

Original: Differently from a similar study carried out by Chen et al.
Revised: Differently from a similar study carried out by Chen et al.

Original: [11],we tuned the hyper-parameter of the sign-agnostic masking in the following range: p ∈{ 0 . 1 , 0 .
Revised: [11], we tuned the hyper-parameter of the sign-agnostic masking in the following range: p ∈ { 0 . 1 , 0 .

Original: The experimental setup complies with the one described in appendix C.1.
Revised: The experimental setup complies with the one described in appendix C.1.

Original: Figure 12, plotting testand validation results for the CelebA dataset [43], shows that the performance of Sign-Agnostic
Revised: Figure 12, plotting test and validation results for the CelebA dataset [44], shows that the performance of Sign-Agnostic

Original: GradDrop closely matches the original algorithm.
Revised: GradDrop closely matches the original algorithm.

Original: Therefore, sign conﬂicts across per-task gradientsdo not seem to play a signiﬁcant role in GradDrop’s performance.[Figure]
Revised: Therefore, sign conﬂicts across per-task gradients do not seem to play a signiﬁcant role in GradDrop’s performance.

Original: E Supplementary Reinforcement Learning Experiments 
Revised: E Supplementary Reinforcement Learning Experiments E.1 Addendum

Original: [Figure] E.1 Addendum

Original: [Figure]

Original: This section presents additional plots for the RL experiments in § 4.2.
Revised: This section presents additional plots for the RL experiments in § 4.2.

Original: Speciﬁcally, Figure 13 re-plots
Revised: Speciﬁcally, Figure 13 re-plots

Original: Figure 4(a) and 4(b) with the omitted IMTL results, while Figure 14 shows the learning curves omitted
Revised: Figure 4(a) and 4(b) with the omitted IMTL results, while Figure 14 shows the learning curves omitted

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: from § 4.2.
Revised: from § 4.2.

Original: As pointed out in § 4.2, none of the IMTL runs successfully terminated due to numericalinstability.
Revised: As pointed out in § 4.2, none of the IMTL runs successfully terminated due to numerical instability.

Original: Indeed, Liu et al.
Revised: Indeed, Liu et al.

Original: [41] show that, in supervised settings, coefﬁcients do not ﬂuctuate muchacross epochs [41, Figure 4, appendix B] and never become negative.
Revised: [42] show that, in supervised settings, coefﬁcients do not ﬂuctuate much across epochs [42, Figure 4, appendix B] and never become negative.

Original: By contrast, up to 50% ofthe scaling coefﬁcients α are negative in our experiments, thus reversing subtask gradient directions.
Revised: By contrast, up to 50% of the scaling coefﬁcients α are negative in our experiments, thus reversing subtask gradient directions.

Original: MGDA, which constrains the weights, is more stable and is comparable to unitary scalarization.
Revised: MGDA, which constrains the weights, is more stable and is comparable to unitary scalarization.

Original: Inorder to avoid incomplete curves and unfair calculations of the mean, Figure 14 plots the highestvalue ever achieved by any seed as a dashed horizontal line.
Revised: In order to avoid incomplete curves and unfair calculations of the mean, Figure 14 plots the highest value ever achieved by any seed as a dashed horizontal line.

Original: The IMTL results in Figure 13, instead,report the best average success rate of each seed until its termination.
Revised: The IMTL results in Figure 13, instead, report the best average success rate of each seed until its termination.

Original: E.2 Ablation studies
Revised: E.2 Ablation studies

Original: Figure 18 presents our ablations for MT10 experiments.
Revised: Figure 18 presents our ablations for MT10 experiments.

Original: Due to computational constraints, we ranablations on the unitary scalarization and PCGrad since these are the two methods previously testedin the RL setting.
Revised: Due to computational constraints, we ran ablations on the unitary scalarization and PCGrad since these are the two methods previously tested in the RL setting.

Original: Figure 15 shows ablation studies on the effect of regularization on MT10 and MT50.
Revised: Figure 15 shows ablation studies on the effect of regularization on MT10 and MT50.

Original: In spite of CIoverlaps, actor l 2 regularization pushes the average higher on both benchmarks, motivating our use ofregularization for the experiments in § 4.2.
Revised: In spite of CI overlaps, actor l 2 regularization pushes the average higher on both benchmarks, motivating our use of regularization for the experiments in § 4.2.

Original: Furthermore, the gap between the averages tends to widenwith the number of updates on MT50, suggesting improved stabilization. 
Revised: Furthermore, the gap between the averages tends to widen with the number of updates on MT50, suggesting improved stabilization. [Figure]

Original: E.3 Sensitivity to Reward Normalization
Revised: E.3 Sensitivity to Reward Normalization

Original: Figure 16 shows that multitask agent performance is highly sensitive to the reward normalizationmoving average hyperparameter 3 motivating our buffer normalization in Section 4.2.
Revised: Figure 16 shows that multitask agent performance is highly sensitive to the reward normalization moving average hyperparameter 4 motivating our buffer normalization in Section 4.2.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: 4 https facebookresearch mtenv 4a6d9d6fdfb321f1b51f890ef36b5161359e972d

Original: [Figure] [Figure]
Revised: [Table] [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure][Table]
Revised: [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Use PCGrad parameters for the baseline (shared alpha)
Revised: Use PCGrad parameters for the baseline (shared alpha)

Original: Increase replay buffer size
Revised: Increase replay buffer size

Original: Normalise reward at the buffer level
Revised: Normalise reward at the buffer level

Original: Regularize the actor
Revised: Regularize the actor

Original: Hyperparameters from [Sodhani et al, 2021]
Revised: Hyperparameters from [Sodhani et al, 2021]