Original: G RADIENT S URGERY FOR M ULTI -T ASK L EARNING
Revised: G RADIENT S URGERY FOR M ULTI -T ASK L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While deep learning and deep reinforcement learning systems have demonstrated impressive results in domains such as image classiﬁcation, game playing, and robotic control, data efﬁciency remains a major challenge, particularly as these algorithms learn individual tasks from scratch.
Revised: While deep learning and deep reinforcement learning systems have demonstrated impressive results in domains such as image classiﬁcation, game playing, and robotic control, data efﬁciency remains a major challenge, particularly as these algorithms learn individual tasks from scratch.

Original: Multi-task learning has emerged as a promising approach for sharing structure across multiple tasks to enable more efﬁcient learning.
Revised: Multi-task learning has emerged as a promising approach for sharing structure across multiple tasks to enable more efﬁcient learning.

Original: However, the multi-task setting presents a number of optimization challenges, making it difﬁcult to realize large efﬁciency gains compared to learning tasks independently.
Revised: However, the multi-task setting presents a number of optimization challenges, making it difﬁcult to realize large efﬁciency gains compared to learning tasks independently.

Original: The reasons why multi-task learning is so challenging compared to single task learning are not fully understood.
Revised: The reasons why multi-task learning is so challenging compared to single task learning are not fully understood.

Original: Motivated by the insight that gradient interference causes optimization challenges, we develop a simple and general approach for avoiding interference between gradients from different tasks, by altering the gradients through a technique we refer to as “gradient surgery”.
Revised: Motivated by the insight that gradient interference causes optimization challenges, we develop a simple and general approach for avoiding interference between gradients from different tasks, by altering the gradients through a technique we refer to as “gradient surgery”.

Original: We propose a form of gradient surgery that projects the gradient of a task onto the normal plane of the gradient of any other task that has a conﬂicting gradient.
Revised: We propose a form of gradient surgery that projects the gradient of a task onto the normal plane of the gradient of any other task that has a conﬂicting gradient.

Original: On a series of challenging multi-task supervised and multi-task reinforcement learning problems, we ﬁnd that this approach leads to substantial gains in efﬁciency and performance.
Revised: On a series of challenging multi-task supervised and multi-task reinforcement learning problems, we ﬁnd that this approach leads to substantial gains in efﬁciency and performance.

Original: Further, it can be effectively combined with previously-proposed multi-task architectures for enhanced performance in a model-agnostic way.
Revised: Further, it can be effectively combined with previously-proposed multi-task architectures for enhanced performance in a model-agnostic way.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: While deep learning and deep reinforcement learning (RL) have shown considerable promise in enabling systems to perform complex tasks, the data requirements of current methods make it difﬁcult to learn a breadth of capabilities particularly when all tasks are learned individually from scratch.
Revised: While deep learning and deep reinforcement learning (RL) have shown considerable promise in enabling systems to perform complex tasks, the data requirements of current methods make it difﬁcult to learn a breadth of capabilities particularly when all tasks are learned individually from scratch.

Original: A natural approach to such multi-task learning problems is to train a single network on all tasks jointly, with the aim of discovering shared structure across the tasks in a way that achieves greater efﬁciency and performance than solving the tasks individually.
Revised: A natural approach to such multi-task learning problems is to train a single network on all tasks jointly, with the aim of discovering shared structure across the tasks in a way that achieves greater efﬁciency and performance than solving the tasks individually.

Original: However, learning multiple tasks all at once results in a difﬁcult optimization problem, sometimes leading to worse overall performance and data efﬁciency compared to learning tasks individually (Parisotto et al., 2015; Rusu et al., 2016a).
Revised: However, learning multiple tasks all at once results in a difﬁcult optimization problem, sometimes leading to worse overall performance and data efﬁciency compared to learning tasks individually (Parisotto et al., 2015; Rusu et al., 2016a).

Original: These optimization challenges are so prevalent that multiple multi-task RL algorithms have considered using independent training as a subroutine of the algorithm before distilling the independent models into a multi-tasking model (Levine et al., 2016; Parisotto et al., 2015;
Revised: These optimization challenges are so prevalent that multiple multi-task RL algorithms have considered using independent training as a subroutine of the algorithm before distilling the independent models into a multi-tasking model (Levine et al., 2016; Parisotto et al., 2015;

Original: Rusu et al., 2016a; Ghosh et al., 2017; Teh et al., 2017), producing a multi-task model but losing out on the efﬁciency gains over independent training.
Revised: Rusu et al., 2016a; Ghosh et al., 2017; Teh et al., 2017), producing a multi-task model but losing out on the efﬁciency gains over independent training.

Original: If we could tackle the optimization challenges of multi-task learning effectively, we may be able to actually realize the hypothesized beneﬁts of multi-task learning without the cost in ﬁnal performance.
Revised: If we could tackle the optimization challenges of multi-task learning effectively, we may be able to actually realize the hypothesized beneﬁts of multi-task learning without the cost in ﬁnal performance.

Original: While there has been a signiﬁcant amount of research in multi-task learning (Caruana, 1997; Ruder, 2017), the optimization challenges are not well understood.
Revised: While there has been a signiﬁcant amount of research in multi-task learning (Caruana, 1997; Ruder, 2017), the optimization challenges are not well understood.

Original: Prior work has described varying learning speeds of different tasks (Chen et al., 2017) and plateaus in the optimization landscape (Schaul et al., 2019) as potential causes, while a range of other works have focused on the model architecture (Misra et al., 2016b; Liu et al., 2018).
Revised: Prior work has described varying learning speeds of different tasks (Chen et al., 2017) and plateaus in the optimization landscape (Schaul et al., 2019) as potential causes, while a range of other works have focused on the model architecture (Misra et al., 2016b; Liu et al., 2018).

Original: In this work, we instead hypothesize that the central optimization issue in multi-task learning arises from gradients from different tasks conﬂicting with one another.
Revised: In this work, we instead hypothesize that the central optimization issue in multi-task learning arises from gradients from different tasks conﬂicting with one another.

Original: In particular, we deﬁne two gradients to be conﬂicting if they point away from one another (i.e., have a negative cosine similarity).
Revised: In particular, we deﬁne two gradients to be conﬂicting if they point away from one another (i.e., have a negative cosine similarity).

Original: As a concrete example, consider the 2D optimization landscapes of two task objectives shown in Figure 1.
Revised: As a concrete example, consider the 2D optimization landscapes of two task objectives shown in Figure 1.

Original: The optimization landscape of each task consists of a deep valley, as has been characterized of neural network optimization landscapes in the past (Goodfellow et al., 2014).
Revised: The optimization landscape of each task consists of a deep valley, as has been characterized of neural network optimization landscapes in the past (Goodfellow et al., 2014).

Original: When considering the combined optimization landscape for multiple tasks, SGD produces gradients that struggle to efﬁciently ﬁnd the optimum.
Revised: When considering the combined optimization landscape for multiple tasks, SGD produces gradients that struggle to efﬁciently ﬁnd the optimum.

Original: This occurs due to a gradient thrashing phenomenon, where the gradient of one task destabilizes optimization in the valley.
Revised: This occurs due to a gradient thrashing phenomenon, where the gradient of one task destabilizes optimization in the valley.

Original: We can observe this in Figure 1(d) when the optimization reaches the deep valley of task 1, but is prevented from traversing the valley to an optimum.
Revised: We can observe this in Figure 1(d) when the optimization reaches the deep valley of task 1, but is prevented from traversing the valley to an optimum.

Original: In Section 6.2, we ﬁnd experimentally that this thrashing phenomenon also occurs in a neural network multi-task learning problem.
Revised: In Section 6.2, we ﬁnd experimentally that this thrashing phenomenon also occurs in a neural network multi-task learning problem.

Original: The core contribution of this work is a method for mitigating gradient interference by altering the gradients directly, i.e. by performing “gradient surgery”.
Revised: The core contribution of this work is a method for mitigating gradient interference by altering the gradients directly, i.e. by performing “gradient surgery”.

Original: If two gradients are conﬂicting, we alter the gradients by projecting each onto the normal plane of the other, preventing the interfering components of the gradient from being applied to the network.
Revised: If two gradients are conﬂicting, we alter the gradients by projecting each onto the normal plane of the other, preventing the interfering components of the gradient from being applied to the network.

Original: We refer to this particular form of gradient surgery as projecting conﬂicting gradients (PCGrad).
Revised: We refer to this particular form of gradient surgery as projecting conﬂicting gradients (PCGrad).

Original: PCGrad is model-agnostic, requiring only a single modiﬁcation to the application of gradients.
Revised: PCGrad is model-agnostic, requiring only a single modiﬁcation to the application of gradients.

Original: Hence, it is easy to apply to a range of problem settings, including multi-task supervised learning and multi-task reinforcement learning, and can also be readily combined with other multi-task learning approaches, such as those that modify the architecture.
Revised: Hence, it is easy to apply to a range of problem settings, including multi-task supervised learning and multi-task reinforcement learning, and can also be readily combined with other multi-task learning approaches, such as those that modify the architecture.

Original: We evaluate PCGrad on multi-task CIFAR classiﬁcation, multi-objective scene understanding, a challenging multi-task RL domain, and goal-conditioned RL.
Revised: We evaluate PCGrad on multi-task CIFAR classiﬁcation, multi-objective scene understanding, a challenging multi-task RL domain, and goal-conditioned RL.

Original: Across the board, we ﬁnd PCGrad leads to signiﬁcant improvements in terms of data efﬁciency, optimization speed, and ﬁnal performance compared to prior approaches.
Revised: Across the board, we ﬁnd PCGrad leads to signiﬁcant improvements in terms of data efﬁciency, optimization speed, and ﬁnal performance compared to prior approaches.

Original: Further, on multi-task supervised learning tasks,
Revised: Further, on multi-task supervised learning tasks,

Original: PCGrad can be successfully combined with prior state-of-the-art methods for multi-task learning for even greater performance.
Revised: PCGrad can be successfully combined with prior state-of-the-art methods for multi-task learning for even greater performance.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: The goal of multi-task learning is to ﬁnd parameters θ of a model f θ that achieve high average performance across all the training tasks drawn from a distribution of tasks p ( T ) .
Revised: The goal of multi-task learning is to ﬁnd parameters θ of a model f θ that achieve high average performance across all the training tasks drawn from a distribution of tasks p ( T ) .

Original: More formally, we aim to solve the problem: min θ E T i ∼ p ( T )
Revised: More formally, we aim to solve the problem: min θ E T i ∼ p ( T )

Original: [ L i ( f θ )] , where L i is a loss function for the i -th task T i that we want to minimize.
Revised: [ L i ( f θ )] , where L i is a loss function for the i -th task T i that we want to minimize.

Original: To obtain a model that solves a speciﬁc task from the task distribution p ( T ) , we deﬁne a task-conditioned model f θ ( y | x, z i ) , with input x , output y , and encoding z i for task T i , which could be provided as a one-hot vector or in any other form.
Revised: To obtain a model that solves a speciﬁc task from the task distribution p ( T ) , we deﬁne a task-conditioned model f θ ( y | x, z i ) , with input x , output y , and encoding z i for task T i , which could be provided as a one-hot vector or in any other form.

Original:  M ULTI -T ASK L EARNING VIA G RADIENT S URGERY
Revised:  M ULTI -T ASK L EARNING VIA G RADIENT S URGERY

Original: While the multi-task problem can in principle be solved by simply applying a standard single-task algorithm with a suitable task identiﬁer provided to the model or a simple multi-head or multi-output model, a number of prior works (Parisotto et al., 2015; Rusu et al., 2016a; Sener & Koltun, 2018) have found this learning problem to be difﬁcult, especially in the reinforcement learning setting.
Revised: While the multi-task problem can in principle be solved by simply applying a standard single-task algorithm with a suitable task identiﬁer provided to the model or a simple multi-head or multi-output model, a number of prior works (Parisotto et al., 2015; Rusu et al., 2016a; Sener & Koltun, 2018) have found this learning problem to be difﬁcult, especially in the reinforcement learning setting.

Original: We hypothesize that one of the main challenges of multi-task learning can be characterized by conﬂicting and thrashing gradients, and ﬁnd that this can signiﬁcantly impede learning progress, especially when combined with iterative data collection.
Revised: We hypothesize that one of the main challenges of multi-task learning can be characterized by conﬂicting and thrashing gradients, and ﬁnd that this can signiﬁcantly impede learning progress, especially when combined with iterative data collection.

Original: We identify possible causes for this problem and propose a simple and general approach to mitigate it.
Revised: We identify possible causes for this problem and propose a simple and general approach to mitigate it.

Original: 3. T HRASHING G RADIENTS IN M ULTI -T ASK O PTIMIZATION L ANDSCAPES
Revised: 3. T HRASHING G RADIENTS IN M ULTI -T ASK O PTIMIZATION L ANDSCAPES

Original: We hypothesize that a key optimization issue in multi-task learning arises when gradients from multiple tasks are in conﬂict with one another, i.e. when gradients point away from one another.
Revised: We hypothesize that a key optimization issue in multi-task learning arises when gradients from multiple tasks are in conﬂict with one another, i.e. when gradients point away from one another.

Original: More speciﬁcally, we hypothesize that such conﬂict may lead to gradient thrashing .
Revised: More speciﬁcally, we hypothesize that such conﬂict may lead to gradient thrashing .

Original: Concretely, gradient thrashing refers to the phenomenon where a large gradient for one task changes the parameter vectors in a way that substantially decreases performance on another task.
Revised: Concretely, gradient thrashing refers to the phenomenon where a large gradient for one task changes the parameter vectors in a way that substantially decreases performance on another task.

Original: Since worse performance typically leads to larger gradients, this results in alternating gradient directions, where, at the next iteration, the second task will have large gradients that dominate and reduce performance on the former task.
Revised: Since worse performance typically leads to larger gradients, this results in alternating gradient directions, where, at the next iteration, the second task will have large gradients that dominate and reduce performance on the former task.

Original: This issue can be particularly pronounced for neural network optimization, since neural network loss landscapes are known to resemble long narrow valleys (Goodfellow et al., 2014), where the gradient perpendicular to the direction of the valley will be small.
Revised: This issue can be particularly pronounced for neural network optimization, since neural network loss landscapes are known to resemble long narrow valleys (Goodfellow et al., 2014), where the gradient perpendicular to the direction of the valley will be small.

Original: We aim to study this hypothesis through two toy examples.
Revised: We aim to study this hypothesis through two toy examples.

Original: First, consider the two-dimensional optimization landscape illustrated in Fig.
Revised: First, consider the two-dimensional optimization landscape illustrated in Fig.

Original: 1a, where the landscape for each task objective corresponds to a deep and curved valley (Fig.
Revised: 1a, where the landscape for each task objective corresponds to a deep and curved valley (Fig.

Original: 1b and 1c).
Revised: 1b and 1c).

Original: The optima of this multi-task objective correspond to where the two valleys meet.
Revised: The optima of this multi-task objective correspond to where the two valleys meet.

Original: More details on the optimization landscape are in Appendix B.
Revised: More details on the optimization landscape are in Appendix B.

Original: We observe that the gradient thrashing hypothesis is consistent with what we observe when running
Revised: We observe that the gradient thrashing hypothesis is consistent with what we observe when running

Original: Adam (Kingma & Ba, 2014) on this landscape in Fig.
Revised: Adam (Kingma & Ba, 2014) on this landscape in Fig.

Original: 1d, where we observe that Adam does not traverse one valley towards the other, preventing it from reaching an optimum.
Revised: 1d, where we observe that Adam does not traverse one valley towards the other, preventing it from reaching an optimum.

Original: We also aim to detect if a similar phenomenon occurs in multi-task learning with a neural network with thousands of parameters on a toy regression problem.
Revised: We also aim to detect if a similar phenomenon occurs in multi-task learning with a neural network with thousands of parameters on a toy regression problem.

Original: To measure the extent of gradient thrashing, we plot the cosine similarity between the gradients of two tasks throughout the beginning of learning in Fig.
Revised: To measure the extent of gradient thrashing, we plot the cosine similarity between the gradients of two tasks throughout the beginning of learning in Fig.

Original: 4 (left).
Revised: 4 (left).

Original: We indeed observe a signiﬁcant level of gradient thrashing at every iteration, where the cosine similarity varies between − 0 .
Revised: We indeed observe a signiﬁcant level of gradient thrashing at every iteration, where the cosine similarity varies between − 0 .

Original: 75 and 0 .
Revised: 75 and 0 .

Original: 75 at a very high frequency.
Revised: 75 at a very high frequency.

Original: Motivated by these observations, we develop an algorithm that aims to alleviate the optimization challenges caused by gradient thrashing by preventing such gradient conﬂict between tasks.
Revised: Motivated by these observations, we develop an algorithm that aims to alleviate the optimization challenges caused by gradient thrashing by preventing such gradient conﬂict between tasks.

Original: 3. PCG RAD : P ROJECTING C ONFLICTING G RADIENTS
Revised: 3. PCG RAD : P ROJECTING C ONFLICTING G RADIENTS

Original: We aim to prevent gradient thrashing by directly altering the gradients themselves, i.e. through “gradient surgery.”
Revised: We aim to prevent gradient thrashing by directly altering the gradients themselves, i.e. through “gradient surgery.”

Original: To be maximally effective and maximally applicable, we must perform surgery in a way that still allows for positive interactions between the task gradients and does not introduce any assumptions on the form of the model.
Revised: To be maximally effective and maximally applicable, we must perform surgery in a way that still allows for positive interactions between the task gradients and does not introduce any assumptions on the form of the model.

Original: We start by ﬁrst detecting whether two gradients are in conﬂict, by measuring whether they point away from one another.
Revised: We start by ﬁrst detecting whether two gradients are in conﬂict, by measuring whether they point away from one another.

Original: More concretely, we characterize two tasks as conﬂicting for the current parameter setting if they yield a negative cosine similarity between their respective gradients.
Revised: More concretely, we characterize two tasks as conﬂicting for the current parameter setting if they yield a negative cosine similarity between their respective gradients.

Original: The goal of PCGrad is to modify the gradients for each task so as to minimize negative conﬂict with other task gradients, which will in turn mitigate gradient thrashing.
Revised: The goal of PCGrad is to modify the gradients for each task so as to minimize negative conﬂict with other task gradients, which will in turn mitigate gradient thrashing.

Original: To deconﬂict gradients during optimization, PCGrad adopts a simple procedure: if the gradients between two tasks are in conﬂict, i.e. their cosine similarity is negative, we project the gradient from one task onto the normal plane of the gradient of the other task.
Revised: To deconﬂict gradients during optimization, PCGrad adopts a simple procedure: if the gradients between two tasks are in conﬂict, i.e. their cosine similarity is negative, we project the gradient from one task onto the normal plane of the gradient of the other task.

Original: This amounts to removing the conﬂicting component of the gradient for the task, thereby reducing the amount of destructive gradient interference between tasks.
Revised: This amounts to removing the conﬂicting component of the gradient for the task, thereby reducing the amount of destructive gradient interference between tasks.

Original: A pictorial description of this idea is shown in Fig.
Revised: A pictorial description of this idea is shown in Fig.

Original: Suppose the gradient for task T i is g i , and the gradient for task
Revised: Suppose the gradient for task T i is g i , and the gradient for task

Original: T j is g j .
Revised: T j is g j .

Original: PCGrad proceeds as follows: (1)
Revised: PCGrad proceeds as follows: (1)

Original: First, it determines whether g i conﬂicts with g j by computing the cosine similarity between vectors g i and g j , where negative values indicate conﬂicting gradients.
Revised: First, it determines whether g i conﬂicts with g j by computing the cosine similarity between vectors g i and g j , where negative values indicate conﬂicting gradients.

Original: (2) If the cosine similarity is negative, we replace g i by its projection onto the normal plane of g j : g i
Revised: (2) If the cosine similarity is negative, we replace g i by its projection onto the normal plane of g j : g i

Original: =
Revised: =

Original: g i − g i · g j (cid:107) g j (cid:107) 

Revised: g i − g i · g j (cid:107) g j (cid:107)

Original: g j .
Revised: g j .

Original: If the gradients are not in conﬂict, i.e. cosine similarity is non-negative, the original gradient g i remains unaltered.
Revised: If the gradients are not in conﬂict, i.e. cosine similarity is non-negative, the original gradient g i remains unaltered.

Original: PCGrad repeats this process across all of the other tasks sampled in random order from the current
Revised: PCGrad repeats this process across all of the other tasks sampled in random order from the current

Original: Require: Current model parameters θ 1: Sample mini-batch of tasks B =
Revised: Require: Current model parameters θ 1: Sample mini-batch of tasks B =

Original: {T k } ∼ p ( T ) 2: for T
Revised: {T k } ∼ p ( T ) 2: for T

Original: i ∼ B in sequence do 3:
Revised: i ∼ B in sequence do 3:

Original: Compute gradient g i of T i as g i = ∇ θ L i ( f θ ) 4: for T
Revised: Compute gradient g i of T i as g i = ∇ θ L i ( f θ ) 4: for T

Original: j uniformly ∼
Revised: j uniformly ∼

Original: B in random order do 5:
Revised: B in random order do 5:

Original: Compute gradient g j of task T j as g j = ∇ θ L j ( f θ ) 6:
Revised: Compute gradient g j of task T j as g j = ∇ θ L j ( f θ ) 6:

Original: Compute cosine similarity between g i as g j as cos( φ ij ) =
Revised: Compute cosine similarity between g i as g j as cos( φ ij ) =

Original: g i · g j (cid:107)
Revised: g i · g j (cid:107)

Original: g i (cid:107)(cid:107) g j (cid:107) .
Revised: g i (cid:107)(cid:107) g j (cid:107) .

Original: 7: if cos( φ ij ) < 0 then 8:
Revised: 7: if cos( φ ij ) < 0 then 8:

Original: Set g i
Revised: Set g i

Original: =
Revised: =

Original: g i − g i · g j (cid:107) g j (cid:107) 

Revised: g i − g i · g j (cid:107) g j (cid:107)

Original: g j // Subtract the projection of g i onto g j 9: end if 10: end for 11:
Revised: g j // Subtract the projection of g i onto g j 9: end if 10: end for 11:

Original: Store g proj i
Revised: Store g proj i

Original: = g i 12: end for 13: return update ∆ θ = (cid:80)
Revised: = g i 12: end for 13: return update ∆ θ = (cid:80)

Original: i g proj
Revised: i g proj

Original: i batch T j ∀ j (cid:54)
Revised: i batch T j ∀ j (cid:54)

Original: =
Revised: =

Original: i , resulting in the gradient g proj i that is applied for task T i .
Revised: i , resulting in the gradient g proj i that is applied for task T i .

Original: We perform the same procedure for all tasks in the batch to obtain their respective gradients.
Revised: We perform the same procedure for all tasks in the batch to obtain their respective gradients.

Original: The full update procedure is described in Algorithm 1 and a discussion on using a random task order is included in Appendix D.
Revised: The full update procedure is described in Algorithm 1 and a discussion on using a random task order is included in Appendix D.

Original: This procedure, while simple to implement, ensures that the gradients that we apply for each task per batch interfere minimally with the other tasks in the batch, mitigating the thrashing gradient problem, producing a variant on standard ﬁrst-order gradient descent in the multi-objective setting.
Revised: This procedure, while simple to implement, ensures that the gradients that we apply for each task per batch interfere minimally with the other tasks in the batch, mitigating the thrashing gradient problem, producing a variant on standard ﬁrst-order gradient descent in the multi-objective setting.

Original: In practice, the PCGrad gradient surgery method can be combined with any gradient-based optimizer, including commonly used methods such as SGD with momentum and Adam (Kingma & Ba, 2014), by simply passing the computed update to the respective optimizer instead of the original gradient.
Revised: In practice, the PCGrad gradient surgery method can be combined with any gradient-based optimizer, including commonly used methods such as SGD with momentum and Adam (Kingma & Ba, 2014), by simply passing the computed update to the respective optimizer instead of the original gradient.

Original: Our experimental results verify the hypothesis that this procedure reduces the problem of thrashing gradients, and ﬁnd that, as a result, learning progress is substantially improved.
Revised: Our experimental results verify the hypothesis that this procedure reduces the problem of thrashing gradients, and ﬁnd that, as a result, learning progress is substantially improved.

Original: Finally, we analyze the convergence of this procedure in Theorem 1 in the two-task setting, to ensure that the procedure is sensible under the standard assumptions in optimization.
Revised: Finally, we analyze the convergence of this procedure in Theorem 1 in the two-task setting, to ensure that the procedure is sensible under the standard assumptions in optimization.

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider two task loss functions L 1 : R n → R and L 2 : R n → R which are convex and differentiable.
Revised: Consider two task loss functions L 1 : R n → R and L 2 : R n → R which are convex and differentiable.

Original: For all θ ∈ R n , let L ( θ ) =
Revised: For all θ ∈ R n , let L ( θ ) =

Original: L 1 ( θ ) + L 2 ( θ ) , i.e. L is a multi-task objective.
Revised: L 1 ( θ ) + L 2 ( θ ) , i.e. L is a multi-task objective.

Original: Let φ be the angle between ∇L 1 ( θ ) and ∇L 2 ( θ ) .
Revised: Let φ be the angle between ∇L 1 ( θ ) and ∇L 2 ( θ ) .

Original: Suppose L is differentiable and that its gradient is
Revised: Suppose L is differentiable and that its gradient is

Original: Lipschitz continuous with constant L > 0 , i.e. we have ||∇L ( θ 1 ) − ∇L ( θ 2 ) ||
Revised: Lipschitz continuous with constant L > 0 , i.e. we have ||∇L ( θ 1 ) − ∇L ( θ 2 ) ||

Original: 2 ≤ L || θ 1 − θ 2 || 2 for any θ 1 , θ 2 .
Revised: 2 ≤ L || θ 1 − θ 2 || 2 for any θ 1 , θ 2 .

Original: Then, the PCGrad update rule with step size t ≤ 1 L will converge to either (1) a location in the optimization landscape where cos( φ ) =
Revised: Then, the PCGrad update rule with step size t ≤ 1 L will converge to either (1) a location in the optimization landscape where cos( φ ) =

Original: − 1 or (2) the optimal value L ( θ ∗ ) .
Revised: − 1 or (2) the optimal value L ( θ ∗ ) .

Original: Proof.
Revised: Proof.

Original: See Appendix A.
Revised: See Appendix A.

Original: Theorem 1 states that application of the PCGrad update in the two-task setting with a convex and
Revised: Theorem 1 states that application of the PCGrad update in the two-task setting with a convex and

Original: Lipschitz multi-task loss function L leads to convergence to either the minimizer of L or a potentially sub-optimal objective value.
Revised: Lipschitz multi-task loss function L leads to convergence to either the minimizer of L or a potentially sub-optimal objective value.

Original: A sub-optimal solution occurs when the cosine similarity between the gradients of the two tasks is − 1 , i.e. the gradients directly conﬂict, leading to zero gradient after applying PCGrad.
Revised: A sub-optimal solution occurs when the cosine similarity between the gradients of the two tasks is − 1 , i.e. the gradients directly conﬂict, leading to zero gradient after applying PCGrad.

Original: However, in practice, since we are using SGD, which is a noisy estimate of the true batch gradients, the cosine similarity between the gradients of two tasks in a minibatch is unlikely to be − 1 , thus avoiding this scenario.
Revised: However, in practice, since we are using SGD, which is a noisy estimate of the true batch gradients, the cosine similarity between the gradients of two tasks in a minibatch is unlikely to be − 1 , thus avoiding this scenario.

Original:  T HE P RACTICAL O PERATIONS OF PCG RAD
Revised:  T HE P RACTICAL O PERATIONS OF PCG RAD

Original: We apply PCGrad to both supervised learning and reinforcement learning problem settings with multiple tasks or goals.
Revised: We apply PCGrad to both supervised learning and reinforcement learning problem settings with multiple tasks or goals.

Original: In this section, we discuss the practical instantiations of PCGrad in those settings.
Revised: In this section, we discuss the practical instantiations of PCGrad in those settings.

Original: Further implementation details are included in Section 6.
Revised: Further implementation details are included in Section 6.

Original: 4. M ULTI -T ASK S UPERVISED L EARNING
Revised: 4. M ULTI -T ASK S UPERVISED L EARNING

Original: In multi-task supervised learning, each task T i ∼ p ( T ) has a corresponding training dataset D i consisting of N i labeled training examples, i.e. D i = { ( x, y ) n } N i n =1 .
Revised: In multi-task supervised learning, each task T i ∼ p ( T ) has a corresponding training dataset D i consisting of N i labeled training examples, i.e. D i = { ( x, y ) n } N i n =1 .

Original: The objective for each task in this supervised setting is then deﬁned as L i ( f θ ) =
Revised: The objective for each task in this supervised setting is then deﬁned as L i ( f θ ) =

Original: E ( x,y )
Revised: E ( x,y )

Original: ∼D i [ − log f θ ( y | x, z i )] , where z i is a one-hot encoding of task T i .
Revised: ∼D i [ − log f θ ( y | x, z i )] , where z i is a one-hot encoding of task T i .

Original: At each training step, we randomly sample a batch of data points B from the whole dataset (cid:83) i D i and then group the sampled data with the same task encoding into small batches denoted as B i for each T i represented in B .
Revised: At each training step, we randomly sample a batch of data points B from the whole dataset (cid:83) i D i and then group the sampled data with the same task encoding into small batches denoted as B i for each T i represented in B .

Original: We denote the set of tasks appearing in B as B T .
Revised: We denote the set of tasks appearing in B as B T .

Original: After sampling, we precompute the gradient of each task in B T as
Revised: After sampling, we precompute the gradient of each task in B T as

Original: [Equation]
Revised: [Equation]

Original: Given the set of precomputed gradients ∇ θ L i ( f θ ) , we also precompute the cosine similarity between all pairs of the gradients in the set.
Revised: Given the set of precomputed gradients ∇ θ L i ( f θ ) , we also precompute the cosine similarity between all pairs of the gradients in the set.

Original: Using the pre-computed gradients and their similarities, we can obtain the PCGrad update by following Algorithm 1, without re-computing task gradients nor backpropagating into the network.
Revised: Using the pre-computed gradients and their similarities, we can obtain the PCGrad update by following Algorithm 1, without re-computing task gradients nor backpropagating into the network.

Original: Since the PCGrad procedure is only modifying the gradients of shared parameters in the optimization step, it is model-agnostic and can be readily applied to any architecture designed for supervised multi-task learning.
Revised: Since the PCGrad procedure is only modifying the gradients of shared parameters in the optimization step, it is model-agnostic and can be readily applied to any architecture designed for supervised multi-task learning.

Original: In Section 6, we combine PCGrad with two state-of-the-art architectures for multi-task learning, which leads to noticeable improvement over their original performance.
Revised: In Section 6, we combine PCGrad with two state-of-the-art architectures for multi-task learning, which leads to noticeable improvement over their original performance.

Original: 4. M ULTI -T ASK AND G OAL -C ONDITIONED R EINFORCEMENT L EARNING
Revised: 4. M ULTI -T ASK AND G OAL -C ONDITIONED R EINFORCEMENT L EARNING

Original: For multi-task reinforcement learning, PCGrad can be readily applied to policy gradient methods by directly updating the computed policy gradient of each task, following Algorithm 1, analogous to the supervised learning setting.
Revised: For multi-task reinforcement learning, PCGrad can be readily applied to policy gradient methods by directly updating the computed policy gradient of each task, following Algorithm 1, analogous to the supervised learning setting.

Original: For actor-critic algorithms, it is also straightforward to apply PCGrad: we simply replace the task gradients for both the actor and the critic by their gradients computed via
Revised: For actor-critic algorithms, it is also straightforward to apply PCGrad: we simply replace the task gradients for both the actor and the critic by their gradients computed via

Original: PCGrad.
Revised: PCGrad.

Original: Hence, PCGrad can be readily incorporated into a variety of model-free RL algorithms.
Revised: Hence, PCGrad can be readily incorporated into a variety of model-free RL algorithms.

Original: When applying PCGrad to goal-conditioned RL, we represent p ( T ) as a distribution of goals and let z i be the encoding of a goal.
Revised: When applying PCGrad to goal-conditioned RL, we represent p ( T ) as a distribution of goals and let z i be the encoding of a goal.

Original: Similar to the multi-task supervised learning setting discussed above,
Revised: Similar to the multi-task supervised learning setting discussed above,

Original: PCGrad may be combined with various architectures designed for multi-task and goal-conditioned
Revised: PCGrad may be combined with various architectures designed for multi-task and goal-conditioned

Original: RL (Fernando et al., 2017; Devin et al., 2016), where PCGrad operates on the gradients of shared parameters, leaving task-speciﬁc parameters untouched.
Revised: RL (Fernando et al., 2017; Devin et al., 2016), where PCGrad operates on the gradients of shared parameters, leaving task-speciﬁc parameters untouched.

Original: In our experiments, we apply PCGrad to the soft actor-critic (SAC) algorithm (Haarnoja et al., 2018), a recently proposed off-policy actor-critic algorithm that has shown signiﬁcant gains in sample efﬁciency and asymptotic performance across many different domains.
Revised: In our experiments, we apply PCGrad to the soft actor-critic (SAC) algorithm (Haarnoja et al., 2018), a recently proposed off-policy actor-critic algorithm that has shown signiﬁcant gains in sample efﬁciency and asymptotic performance across many different domains.

Original: In SAC, we employ a Qlearning style gradient to compute the gradient of the Q-function network, Q φ ( s, a, z i ) , often known as the critic, and a reparameterization-style gradient to compute the gradient of the policy network π θ ( a | s, z i ) , often known as the actor.
Revised: In SAC, we employ a Qlearning style gradient to compute the gradient of the Q-function network, Q φ ( s, a, z i ) , often known as the critic, and a reparameterization-style gradient to compute the gradient of the policy network π θ ( a | s, z i ) , often known as the actor.

Original: For sampling, we instantiate a set of replay buffers {D i } T i ∼ p ( T ) .
Revised: For sampling, we instantiate a set of replay buffers {D i } T i ∼ p ( T ) .

Original: Training and data collection are alternated throughout training.
Revised: Training and data collection are alternated throughout training.

Original: During a data collection step, we run the policy π θ on all the tasks T i ∼ p ( T ) to collect an equal number of paths for each task and store the paths of each task T i into the corresponding replay buffer D i .
Revised: During a data collection step, we run the policy π θ on all the tasks T i ∼ p ( T ) to collect an equal number of paths for each task and store the paths of each task T i into the corresponding replay buffer D i .

Original: At each training step, we sample an equal amount of data from each replay buffer D
Revised: At each training step, we sample an equal amount of data from each replay buffer D

Original: i to form a stratiﬁed batch.
Revised: i to form a stratiﬁed batch.

Original: For each task T i ∼ p ( T ) , the parameters of the critic θ are optimized to minimize the soft Bellman residual:
Revised: For each task T i ∼ p ( T ) , the parameters of the critic θ are optimized to minimize the soft Bellman residual:

Original: [Equation]
Revised: [Equation]

Original: where γ is the discount factor, ¯ φ are the delayed parameters, and α is a learnable temperature that automatically adjusts the weight of the entropy term.
Revised: where γ is the discount factor, ¯ φ are the delayed parameters, and α is a learnable temperature that automatically adjusts the weight of the entropy term.

Original: For each task T i ∼ p ( T ) , the parameters of the policy π θ are trained to minimize the following objective
Revised: For each task T i ∼ p ( T ) , the parameters of the policy π θ are trained to minimize the following objective

Original: [Equation]
Revised: [Equation]

Original: We compute ∇ φ J ( i ) Q ( φ ) and ∇ θ J ( i )
Revised: We compute ∇ φ J ( i ) Q ( φ ) and ∇ θ J ( i )

Original: π ( θ ) for all T
Revised: π ( θ ) for all T

Original: i ∼ p ( T ) and apply PCGrad to both following
Revised: i ∼ p ( T ) and apply PCGrad to both following

Original: Algorithm 1.
Revised: Algorithm 1.

Original: In the context of SAC speciﬁcally, we further study how the temperature α should be adjusted.
Revised: In the context of SAC speciﬁcally, we further study how the temperature α should be adjusted.

Original: If we use a single learnable temperature for adjusting entropy of the multi-task policy π θ ( a | s, z i ) , SAC may stop exploring once all easier tasks are solved, leading to poor performance on tasks that are harder or require more exploration.
Revised: If we use a single learnable temperature for adjusting entropy of the multi-task policy π θ ( a | s, z i ) , SAC may stop exploring once all easier tasks are solved, leading to poor performance on tasks that are harder or require more exploration.

Original: To address this issue, we propose to learn the temperature on a per-task basis, i.e. using a parametrized model to represent α ψ ( z i ) (which we abbreviate as PA for per-task alpha).
Revised: To address this issue, we propose to learn the temperature on a per-task basis, i.e. using a parametrized model to represent α ψ ( z i ) (which we abbreviate as PA for per-task alpha).

Original: This allows the method to control the entropy of π θ ( a | s, z i ) per-task.
Revised: This allows the method to control the entropy of π θ ( a | s, z i ) per-task.

Original: We optimize the parameters of α ψ ( z i ) using the same constrained optimization framework as in Haarnoja et al.
Revised: We optimize the parameters of α ψ ( z i ) using the same constrained optimization framework as in Haarnoja et al.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Algorithms for multi-task learning typically consider how to train a single model that can solve a variety of different tasks (Caruana, 1997; Bakker & Heskes, 2003; Ruder, 2017).
Revised: Algorithms for multi-task learning typically consider how to train a single model that can solve a variety of different tasks (Caruana, 1997; Bakker & Heskes, 2003; Ruder, 2017).

Original: The multi-task formulation has been applied to many different settings, including supervised learning (Zhang et al., 2014; Long & Wang, 2015; Yang & Hospedales, 2016; Sener & Koltun, 2018; Zamir et al., 2018) and reinforcement-learning (Espeholt et al., 2018; Wilson et al., 2007), as well as many different domains, such as vision (Bilen & Vedaldi, 2016; Misra et al., 2016a; Kokkinos, 2017;
Revised: The multi-task formulation has been applied to many different settings, including supervised learning (Zhang et al., 2014; Long & Wang, 2015; Yang & Hospedales, 2016; Sener & Koltun, 2018; Zamir et al., 2018) and reinforcement-learning (Espeholt et al., 2018; Wilson et al., 2007), as well as many different domains, such as vision (Bilen & Vedaldi, 2016; Misra et al., 2016a; Kokkinos, 2017;

Original: Liu et al., 2018;
Revised: Liu et al., 2018;

Original: Zamir et al., 2018), language (Collobert & Weston, 2008; Dong et al., 2015; McCann et al., 2018;
Revised: Zamir et al., 2018), language (Collobert & Weston, 2008; Dong et al., 2015; McCann et al., 2018;

Original: Radford et al., 2019) and robotics (Riedmiller et al., 2018; Wulfmeier et al., 2019; Hausman et al., 2018).
Revised: Radford et al., 2019) and robotics (Riedmiller et al., 2018; Wulfmeier et al., 2019; Hausman et al., 2018).

Original: While multi-task learning has the promise of accelerating acquisition of large task repertoires, in practice it presents a challenging optimization problem, which has been tackled in several ways in prior work.
Revised: While multi-task learning has the promise of accelerating acquisition of large task repertoires, in practice it presents a challenging optimization problem, which has been tackled in several ways in prior work.

Original: A number of architectural solutions have been proposed to the multi-task learning problem based on multiple modules or paths (Fernando et al., 2017; Devin et al., 2016; Misra et al., 2016b; Rusu et al., 2016b; Rosenbaum et al., 2018; Vandenhende et al., 2019; Rosenbaum et al., 2018), or using attention-based architectures (Liu et al., 2018; Maninis et al., 2019).
Revised: A number of architectural solutions have been proposed to the multi-task learning problem based on multiple modules or paths (Fernando et al., 2017; Devin et al., 2016; Misra et al., 2016b; Rusu et al., 2016b; Rosenbaum et al., 2018; Vandenhende et al., 2019; Rosenbaum et al., 2018), or using attention-based architectures (Liu et al., 2018; Maninis et al., 2019).

Original: Our work is agnostic to the model architecture and can be combined with prior architectural approaches in a complementary fashion.
Revised: Our work is agnostic to the model architecture and can be combined with prior architectural approaches in a complementary fashion.

Original: A different set of multi-task learning approaches aim to decompose the problem into multiple local problems, often corresponding to each task, that are signiﬁcantly easier to learn, akin to divide and conquer algorithms (Levine et al., 2016; Rusu et al., 2016a; Parisotto et al., 2015;
Revised: A different set of multi-task learning approaches aim to decompose the problem into multiple local problems, often corresponding to each task, that are signiﬁcantly easier to learn, akin to divide and conquer algorithms (Levine et al., 2016; Rusu et al., 2016a; Parisotto et al., 2015;

Original: Teh et al., 2017;
Revised: Teh et al., 2017;

Original: Ghosh et al., 2017; Czarnecki et al., 2019).
Revised: Ghosh et al., 2017; Czarnecki et al., 2019).

Original: Eventually, the local models are combined into a single, multi-task policy using different distillation techniques (outlined in (Hinton et al., 2015; Czarnecki et al., 2019)).
Revised: Eventually, the local models are combined into a single, multi-task policy using different distillation techniques (outlined in (Hinton et al., 2015; Czarnecki et al., 2019)).

Original: In contrast to these methods, we propose a simple and cogent scheme for multi-task learning that allows us to learn the tasks simultaneously using a single, shared model without the need for network distillation.
Revised: In contrast to these methods, we propose a simple and cogent scheme for multi-task learning that allows us to learn the tasks simultaneously using a single, shared model without the need for network distillation.

Original: Similarly to our work, a number of prior approaches have observed the difﬁculty of optimization in the multi-task learning setting (Hessel et al., 2019; Chen et al., 2018; Kendall et al., 2018b; Schaul et al., 2019).
Revised: Similarly to our work, a number of prior approaches have observed the difﬁculty of optimization in the multi-task learning setting (Hessel et al., 2019; Chen et al., 2018; Kendall et al., 2018b; Schaul et al., 2019).

Original: Our work, in contrast to many of these optimization schemes, suggests that the challenge in multi-task learning may be attributed to the problem of gradient thrashing, which we address directly by introducing a simple and practical algorithm that de-conﬂicts gradients from different tasks.
Revised: Our work, in contrast to many of these optimization schemes, suggests that the challenge in multi-task learning may be attributed to the problem of gradient thrashing, which we address directly by introducing a simple and practical algorithm that de-conﬂicts gradients from different tasks.

Original: Prior work (Sener & Koltun, 2018) alternatively proposes a gradient-based multi-objective optimization problem for multi-task learning to address the problem of optimizing possibly conﬂicting objectives.
Revised: Prior work (Sener & Koltun, 2018) alternatively proposes a gradient-based multi-objective optimization problem for multi-task learning to address the problem of optimizing possibly conﬂicting objectives.

Original: As noted in Alg 2 in (Sener & Koltun, 2018), it learns a constant scaling factor for per-task gradient to avoid conﬂicting, while our method corrects both the scaling factor and the direction of per-task gradient, which can more effectively deconﬂict gradients.
Revised: As noted in Alg 2 in (Sener & Koltun, 2018), it learns a constant scaling factor for per-task gradient to avoid conﬂicting, while our method corrects both the scaling factor and the direction of per-task gradient, which can more effectively deconﬂict gradients.

Original: Prior work has also used the cosine similarity between gradients to deﬁne when an auxiliary task might be useful for single-task learning (Du et al., 2018).
Revised: Prior work has also used the cosine similarity between gradients to deﬁne when an auxiliary task might be useful for single-task learning (Du et al., 2018).

Original: We similarly use cosine similarity between gradients to determine if the gradients between a pair of tasks are in conﬂict.
Revised: We similarly use cosine similarity between gradients to determine if the gradients between a pair of tasks are in conﬂict.

Original: Unlike Du et al.
Revised: Unlike Du et al.

Original: (2018), we use this measure of gradient conﬂict as a part of gradient surgery in the context of multi-task learning applications.
Revised: (2018), we use this measure of gradient conﬂict as a part of gradient surgery in the context of multi-task learning applications.

Original: A number of works in continual learning have studied how to make gradient updates that do not adversely affect other tasks by projecting the gradients into a space that do not conﬂict with previous tasks (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018).
Revised: A number of works in continual learning have studied how to make gradient updates that do not adversely affect other tasks by projecting the gradients into a space that do not conﬂict with previous tasks (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018).

Original: Those methods focus on the continual learning setting, and either need to solve for the gradient projections using quadratic programming (Lopez-Paz & Ranzato, 2017), or only projecting the gradient onto the normal plane of the average of the gradients of past tasks (Chaudhry et al., 2018).
Revised: Those methods focus on the continual learning setting, and either need to solve for the gradient projections using quadratic programming (Lopez-Paz & Ranzato, 2017), or only projecting the gradient onto the normal plane of the average of the gradients of past tasks (Chaudhry et al., 2018).

Original: In contrast, our work focuses on multi-task learning, does not require solving any QP, and iteratively projects the gradients of each task onto the normal plane of the gradients of each of the other tasks instead of averaging .
Revised: In contrast, our work focuses on multi-task learning, does not require solving any QP, and iteratively projects the gradients of each task onto the normal plane of the gradients of each of the other tasks instead of averaging .

Original: Finally, our method is distinct from and solves a different problem than the projected gradient method (Calamai & Mor´e, 1987), which is an approach for constrained optimization that projects gradients onto the constraint manifold.
Revised: Finally, our method is distinct from and solves a different problem than the projected gradient method (Calamai & Mor´e, 1987), which is an approach for constrained optimization that projects gradients onto the constraint manifold.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The goal of our experiments is to study the following questions: (1) Are conﬂicting gradients a major factor in making optimization for multi-task learning challenging?
Revised: The goal of our experiments is to study the following questions: (1) Are conﬂicting gradients a major factor in making optimization for multi-task learning challenging?

Original: (2) Does PCGrad make the optimization problems easier for various multi-task learning problems including supervised, reinforcement, and goal-conditioned reinforcement learning settings across different task families?
Revised: (2) Does PCGrad make the optimization problems easier for various multi-task learning problems including supervised, reinforcement, and goal-conditioned reinforcement learning settings across different task families?

Original: Can PCGrad be combined with other multi-task learning approaches to further improve performance?
Revised: Can PCGrad be combined with other multi-task learning approaches to further improve performance?

Original: 6. E XPERIMENTAL S ETUP
Revised: 6. E XPERIMENTAL S ETUP

Original: To evaluate our method experimentally, we consider both a multi-task supervised learning and a multi-task reinforcement learning problem setup.
Revised: To evaluate our method experimentally, we consider both a multi-task supervised learning and a multi-task reinforcement learning problem setup.

Original: For supervised learning, we ﬁrst consider the
Revised: For supervised learning, we ﬁrst consider the

Original: MultiMNIST dataset (Sener & Koltun, 2018), which contains two tasks: classifying the digit on the top left and on the bottom right in an overlaid image.
Revised: MultiMNIST dataset (Sener & Koltun, 2018), which contains two tasks: classifying the digit on the top left and on the bottom right in an overlaid image.

Original: Beyond digit classiﬁcation, we also use the CIFAR-100 dataset (Krizhevsky et al., 2009) where each of the 20 label superclasses are treated as distinct tasks, following Rosenbaum et al.
Revised: Beyond digit classiﬁcation, we also use the CIFAR-100 dataset (Krizhevsky et al., 2009) where each of the 20 label superclasses are treated as distinct tasks, following Rosenbaum et al.

Original: We also conduct experiments on the NYUv dataset (Silberman et al., 2012), which consists of RGB-D indoor scene images.
Revised: We also conduct experiments on the NYUv dataset (Silberman et al., 2012), which consists of RGB-D indoor scene images.

Original: Following Liu et al.
Revised: Following Liu et al.

Original: (2018), we evaluate our method on 3 tasks: 13-class semantic segmentation, depth estimation, and surface normal prediction.
Revised: (2018), we evaluate our method on 3 tasks: 13-class semantic segmentation, depth estimation, and surface normal prediction.

Original: In the case of multi-task reinforcement learning, we evaluate our algorithm on the recently proposed Meta-World benchmark (Yu et al., 2019).
Revised: In the case of multi-task reinforcement learning, we evaluate our algorithm on the recently proposed Meta-World benchmark (Yu et al., 2019).

Original: This benchmark includes a variety of simulated robotic manipulation tasks contained in a shared, table-top environment with a simulated
Revised: This benchmark includes a variety of simulated robotic manipulation tasks contained in a shared, table-top environment with a simulated

Original: Sawyer arm (visualized as the ”Push” environment in Fig. 3).
Revised: Sawyer arm (visualized as the ”Push” environment in Fig. 3).

Original: In particular, we use the multi-task benchmarks MT10 and MT50, which consists of the 10 tasks and 50 tasks respectively depicted in
Revised: In particular, we use the multi-task benchmarks MT10 and MT50, which consists of the 10 tasks and 50 tasks respectively depicted in

Original: Fig.
Revised: Fig.

Original: 3 that require diverse strategies to solve them, which makes them difﬁcult to optimize jointly with a single policy.
Revised: 3 that require diverse strategies to solve them, which makes them difﬁcult to optimize jointly with a single policy.

Original: Note that MT10 is a subset of MT50.
Revised: Note that MT10 is a subset of MT50.

Original: To evaluate goal-conditioned RL scenarios, we consider goal-conditioned robotic pushing with a Sawyer robot.
Revised: To evaluate goal-conditioned RL scenarios, we consider goal-conditioned robotic pushing with a Sawyer robot.

Original: This domain is representative of challenges in learning goal-conditioned policies over a wide distribution of goals.
Revised: This domain is representative of challenges in learning goal-conditioned policies over a wide distribution of goals.

Original: For details on the experimental set-up and model architectures see Appendix E.
Revised: For details on the experimental set-up and model architectures see Appendix E.

Original: 6. A NALYSIS OF C ONFLICTING G RADIENTS
Revised: 6. A NALYSIS OF C ONFLICTING G RADIENTS

Original: To answer question (1), we consider a simple regression problem, where each task is regressing the input to the output of a sine function.
Revised: To answer question (1), we consider a simple regression problem, where each task is regressing the input to the output of a sine function.

Original: The amplitude and the phase of each task are varied.
Revised: The amplitude and the phase of each task are varied.

Original: We construct 10 tasks with the amplitude uniformly sampled from the range [0 , 5] and the phase uniformly sampled from the range [0 , π ] .
Revised: We construct 10 tasks with the amplitude uniformly sampled from the range [0 , 5] and the phase uniformly sampled from the range [0 , π ] .

Original: The input is also uniformly sampled from the range [0 , 5] and is concatenated with the one-hot task encoding.
Revised: The input is also uniformly sampled from the range [0 , 5] and is concatenated with the one-hot task encoding.

Original: For training, we use a 3-layer fully-connected neural network with 100 hidden units.
Revised: For training, we use a 3-layer fully-connected neural network with 100 hidden units.

Original: We compare the performance of the network trained with Adam and the network trained with Adam with PCGrad-modiﬁed gradients while plotting the cosine similarity between a pair of tasks during training as shown in Figure 4.
Revised: We compare the performance of the network trained with Adam and the network trained with Adam with PCGrad-modiﬁed gradients while plotting the cosine similarity between a pair of tasks during training as shown in Figure 4.

Original: The plot on the left in Figure 4 demonstrates that the cosine similarity of Adam gradients between a pair of tasks has high variance, which leads to the gradient thrashing problem, while the cosine similarity of the gradient projected by PCGrad yields positive values diminishing the conﬂicting-gradients problem.
Revised: The plot on the left in Figure 4 demonstrates that the cosine similarity of Adam gradients between a pair of tasks has high variance, which leads to the gradient thrashing problem, while the cosine similarity of the gradient projected by PCGrad yields positive values diminishing the conﬂicting-gradients problem.

Original: As shown in the plot on the right in Figure 4, Adam with PCGrad leads to faster learning over Adam, which implies that gradient thrashing is indeed a problem in multi-task optimization and reducing it can result in considerable performance boost.
Revised: As shown in the plot on the right in Figure 4, Adam with PCGrad leads to faster learning over Adam, which implies that gradient thrashing is indeed a problem in multi-task optimization and reducing it can result in considerable performance boost.

Original: 6. M ULTI -T ASK AND M ULTI -O BJECTIVE S UPERVISED L EARNING WITH PCG RAD
Revised: 6. M ULTI -T ASK AND M ULTI -O BJECTIVE S UPERVISED L EARNING WITH PCG RAD

Original: To answer question (3), we perform experiments on three standard multi-task supervised learning datasets: MultiMNIST, multi-task CIFAR-100 and NYUv2.
Revised: To answer question (3), we perform experiments on three standard multi-task supervised learning datasets: MultiMNIST, multi-task CIFAR-100 and NYUv2.

Original: We include the results on MultiMNIST in Appendix C.
Revised: We include the results on MultiMNIST in Appendix C.

Original: For CIFAR-100, we follow (Rosenbaum et al., 2018) to treat 20 coarse labels in the dataset as distinct tasks and create a dataset with 20 tasks and 2500 training instances as well as 500 test instances per task.
Revised: For CIFAR-100, we follow (Rosenbaum et al., 2018) to treat 20 coarse labels in the dataset as distinct tasks and create a dataset with 20 tasks and 2500 training instances as well as 500 test instances per task.

Original: We combine PCGrad with a powerful multi-task learning architecture, routing networks (Rosenbaum et al., 2018; 2019), by simply projecting gradients of the shared parameters in routing networks.
Revised: We combine PCGrad with a powerful multi-task learning architecture, routing networks (Rosenbaum et al., 2018; 2019), by simply projecting gradients of the shared parameters in routing networks.

Original: As shown in Table 1,  combining PCGrad with routing networks leads to a 4 . 5% absolute improvement in test accuracy averaged over 3 runs, while also outperforming other approaches such as training independent networks for each task and cross-stitch (Misra et al., 2016b).
Revised: As shown in Table 1, applying PCGrad to a single network achieves 71% classiﬁcation accuracy, which outperforms most of the prior methods such as independent training and cross-stitch (Misra et al., 2016b). Though routing networks achieve better performance than PCGrad on its own, PCGrad is complementary to routing networks and combining PCGrad with routing networks leads to a 2 . 8% absolute improvement in test accuracy averaged over 3 runs.

Original: We also combine PCGrad with another state-of-art multi-task learning algorithm, MTAN (Liu et al., 2018), and evaluate the performance on a more challenging indoor scene dataset, NYUv2, which contains 3 tasks as described in Section 6.1.
Revised: We also combine PCGrad with another state-of-art multi-task learning algorithm, MTAN (Liu et al., 2018), and evaluate the performance on a more challenging indoor scene dataset, NYUv2, which contains 3 tasks as described in Section 6.1.

Original: We compare MTAN with PCGrad to a list of methods mentioned in Section 6.1, where each method is trained with three different weighting schemes as in (Liu et al., 2018), equal weighting, weight uncertainty (Kendall et al., 2018a), and DWA (Liu et al., 2018).
Revised: We compare MTAN with PCGrad to a list of methods mentioned in Section 6.1, where each method is trained with three different weighting schemes as in (Liu et al., 2018), equal weighting, weight uncertainty (Kendall et al., 2018a), and DWA (Liu et al., 2018).

Original: We only run MTAN with PCGrad with weight uncertainty as we ﬁnd weight uncertainty as the most effective scheme for training MTAN.
Revised: We only run MTAN with PCGrad with weight uncertainty as we ﬁnd weight uncertainty as the most effective scheme for training MTAN.

Original: The results comparing Cross-Stitch, MTAN and
Revised: The results comparing Cross-Stitch, MTAN and

Original: MTAN + PCGrad are presented in Table 2 while the full comparison can be found in Table 4 in the
Revised: MTAN + PCGrad are presented in Table 2 while the full comparison can be found in Table 4 in the

Original: Appendix E.3.
Revised: Appendix E.3.

Original: MTAN with PCGrad is able to achieve the best scores in 8 out of the 9 categories where there are 3 categories per task.
Revised: MTAN with PCGrad is able to achieve the best scores in 8 out of the 9 categories where there are 3 categories per task.

Original: Our multi-task supervised learning results demonstrate that PCGrad can be seamlessly combined with state-of-art multi-task learning architectures and further improve their results on established supervised multi-task learning benchmarks.
Revised: Our multi-task supervised learning results demonstrate that PCGrad can be seamlessly combined with state-of-art multi-task learning architectures and further improve their results on established supervised multi-task learning benchmarks.

Original: 6. M ULTI -T ASK R EINFORCEMENT L EARNING
Revised: 6. M ULTI -T ASK R EINFORCEMENT L EARNING

Original: To answer question (2), we test all methods on 10 and 50 manipulation tasks respectively shown in
Revised: To answer question (2), we test all methods on 10 and 50 manipulation tasks respectively shown in

Original: Figure 3.
Revised: Figure 3.

Original: At each data collection step, we collect 600 samples for each task, and at each training step,  we sample 128 datapoints per task from corresponding replay buffers.
Revised: At each data collection step, we collect 600 samples for each task, and at each training step, [Table] [Table] we sample 128 datapoints per task from corresponding replay buffers.

Original: The results are shown in the two plots on the left in Figure 5.
Revised: The results are shown in the two plots on the left in Figure 5.

Original: We measure success according to the metrics used in the Meta-World[Table] [Table] benchmark where the reported the success rates are averaged across tasks.
Revised: We measure success according to the metrics used in the Meta-World benchmark where the reported the success rates are averaged across tasks.

Original: For all methods, we apply
Revised: For all methods, we apply

Original: PA as discussed in Section 4 to learn a separate alpha term per task as the task encoding in MT and MT50 is just a one-hot encoding.
Revised: PA as discussed in Section 4 to learn a separate alpha term per task as the task encoding in MT and MT50 is just a one-hot encoding.

Original: PCGrad combined with SAC learns all tasks with the best data efﬁciency and successfully solves all of the 10 tasks in MT10 and about 70 % of the 50 tasks in MT50.
Revised: PCGrad combined with SAC learns all tasks with the best data efﬁciency and successfully solves all of the 10 tasks in MT10 and about 70 % of the 50 tasks in MT50.

Original: Training a single SAC policy and a multi-head policy turns out to be unable to acquire half of the skills in both MT10 and MT50, suggesting that eliminating gradient interference across tasks can signiﬁcantly boost performance of multi-task RL.
Revised: Training a single SAC policy and a multi-head policy turns out to be unable to acquire half of the skills in both MT10 and MT50, suggesting that eliminating gradient interference across tasks can signiﬁcantly boost performance of multi-task RL.

Original: Training independent SAC agents is able to eventually solve all tasks in MT10 and 70 % of the tasks in MT50, but requires about 2 millions and 15 millions more samples than PCGrad with SAC in MT10 and MT50 respectively, implying that applying PCGrad can result in leveraging shared structure among tasks that expedites multi-task learning.
Revised: Training independent SAC agents is able to eventually solve all tasks in MT10 and 70 % of the tasks in MT50, but requires about 2 millions and 15 millions more samples than PCGrad with SAC in MT10 and MT50 respectively, implying that applying PCGrad can result in leveraging shared structure among tasks that expedites multi-task learning.

Original: As noted by Yu et al.
Revised: As noted by Yu et al.

Original: , these tasks involve fairly distinct behavior motions, which makes learning all of them with a single policy challenging as demonstrated by poor baseline performance.
Revised: , these tasks involve fairly distinct behavior motions, which makes learning all of them with a single policy challenging as demonstrated by poor baseline performance.

Original: The ability to learn these tasks together opens the door for a number of interesting extensions to meta-learning, goal conditioned RL and generalization to novel task families.
Revised: The ability to learn these tasks together opens the door for a number of interesting extensions to meta-learning, goal conditioned RL and generalization to novel task families.

Original: We present the results of PCGrad on goal-conditioned RL in the following subsection.
Revised: We present the results of PCGrad on goal-conditioned RL in the following subsection.

Original: We also provide an ablation study on the importance of correcting the gradient direction and scaling the gradient magnitudes in PCGrad.
Revised: We also provide an ablation study on the importance of correcting the gradient direction and scaling the gradient magnitudes in PCGrad.

Original: We construct two variants of PCGrad: (1) only applying the gradient direction corrected with PCGrad while keeping the gradient magnitude unchanged and (2) only applying the gradient magnitude computed by PCGrad while keeping the gradient direction unchanged.
Revised: We construct two variants of PCGrad: (1) only applying the gradient direction corrected with PCGrad while keeping the gradient magnitude unchanged and (2) only applying the gradient magnitude computed by PCGrad while keeping the gradient direction unchanged.

Original: As shown in the plot on the left in Figure 6, both variants perform worse than PCGrad and the variant where we only vary the gradient magnitudes is much worse than PCGrad.
Revised: As shown in the plot on the left in Figure 6, both variants perform worse than PCGrad and the variant where we only vary the gradient magnitudes is much worse than PCGrad.

Original: We also compare
Revised: We also compare

Original: PCGrad to a prior method GradNorm (Chen et al., 2018), which scales the magnitude of gradients of all the tasks.
Revised: PCGrad to a prior method GradNorm (Chen et al., 2018), which scales the magnitude of gradients of all the tasks.

Original: As shown in the plot on the right in Figure 6, PCGrad signiﬁcantly outperforms
Revised: As shown in the plot on the right in Figure 6, PCGrad signiﬁcantly outperforms

Original: GradNorm.
Revised: GradNorm.

Original: We also notice that the variant of PCGrad where only the gradient magnitudes change gets comparable results to GradNorm, which suggests that its important to modify both the gradient directions and magnitudes to eliminate interference and achieve good multi-task learning results.
Revised: We also notice that the variant of PCGrad where only the gradient magnitudes change gets comparable results to GradNorm, which suggests that its important to modify both the gradient directions and magnitudes to eliminate interference and achieve good multi-task learning results.

Original: 6. G OAL -C ONDITIONED R EINFORCEMENT L EARNING
Revised: 6. G OAL -C ONDITIONED R EINFORCEMENT L EARNING

Original: For our goal-conditioned RL evaluation, we use the robot-pushing environment described in Sec. 6.
Revised: For our goal-conditioned RL evaluation, we use the robot-pushing environment described in Sec. 6.

Original: where the goals are represented as the concatenations of the initial positions of the puck to be pushed and the its goal location, both of which are uniformly sampled (details in Appendix E.2).
Revised: where the goals are represented as the concatenations of the initial positions of the puck to be pushed and the its goal location, both of which are uniformly sampled (details in Appendix E.2).

Original: We also apply PA as discussed in Section 4 to predict the temperature for entropy term given the goal.
Revised: We also apply PA as discussed in Section 4 to predict the temperature for entropy term given the goal.

Original: We summarize the results in the plot on the right in Figure 5.
Revised: We summarize the results in the plot on the right in Figure 5.

Original: PCGrad with SAC and PA achieves the best performance in terms of average distance to the goal position, while PCGrad with SAC improves over the baseline and a vanilla SAC agent is struggling to successfully accomplish the task.
Revised: PCGrad with SAC and PA achieves the best performance in terms of average distance to the goal position, while PCGrad with SAC improves over the baseline and a vanilla SAC agent is struggling to successfully accomplish the task.

Original: This suggests that PCGrad is able to ease the RL optimization problem also when the task distribution is continuous.
Revised: This suggests that PCGrad is able to ease the RL optimization problem also when the task distribution is continuous.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we identiﬁed one of the major challenges in multi-task optimization: conﬂicting gradients across tasks.
Revised: In this work, we identiﬁed one of the major challenges in multi-task optimization: conﬂicting gradients across tasks.

Original: We proposed a simple algorithm (PCGrad) to mitigate the challenge of conﬂicting gradients via “gradient surgery”.
Revised: We proposed a simple algorithm (PCGrad) to mitigate the challenge of conﬂicting gradients via “gradient surgery”.

Original: PCGrad provides a simple way to project gradients to be orthogonal in a multi-task setting, which substantially improves optimization performance, since the task gradients are prevented from negating each other.
Revised: PCGrad provides a simple way to project gradients to be orthogonal in a multi-task setting, which substantially improves optimization performance, since the task gradients are prevented from negating each other.

Original: We provide some simple didactic examples and analysis of how this procedure works in simple settings, and subsequently show signiﬁcant improvement in optimization for a variety of multi-task supervised learning and reinforcement learning problems.
Revised: We provide some simple didactic examples and analysis of how this procedure works in simple settings, and subsequently show signiﬁcant improvement in optimization for a variety of multi-task supervised learning and reinforcement learning problems.

Original: We show that, once some of the optimization challenges of multi-task learning are alleviated by PCGrad, we can obtain the hypothesized beneﬁts in efﬁciency and asymptotic performance that are believed to be possible in multi-task settings.
Revised: We show that, once some of the optimization challenges of multi-task learning are alleviated by PCGrad, we can obtain the hypothesized beneﬁts in efﬁciency and asymptotic performance that are believed to be possible in multi-task settings.

Original: While we studied multi-task supervised learning and multi-task reinforcement learning in this work, we suspect the problem of conﬂicting gradients to be prevalent in a range of other settings and applications, such as meta-learning, continual learning, multi-goal imitation learning (Codevilla et al., 2018), and multi-task problems in natural language processing applications (McCann et al., 2018).
Revised: While we studied multi-task supervised learning and multi-task reinforcement learning in this work, we suspect the problem of conﬂicting gradients to be prevalent in a range of other settings and applications, such as meta-learning, continual learning, multi-goal imitation learning (Codevilla et al., 2018), and multi-task problems in natural language processing applications (McCann et al., 2018).

Original: Due to its simplicity and model-agnostic nature, we expect that applying PCGrad in these domains to be a promising avenue for future investigation.
Revised: Due to its simplicity and model-agnostic nature, we expect that applying PCGrad in these domains to be a promising avenue for future investigation.

Original: Further, the general idea of gradient surgery may be an important ingredient for alleviating a broader class of optimization challenges in deep learning, such as the challenges in the stability challenges in two-player games (Roth et al., 2017) and multi-agent optimizations (Nedic & Ozdaglar, 2009).
Revised: Further, the general idea of gradient surgery may be an important ingredient for alleviating a broader class of optimization challenges in deep learning, such as the challenges in the stability challenges in two-player games (Roth et al., 2017) and multi-agent optimizations (Nedic & Ozdaglar, 2009).

Original: We believe this work to be a step towards simple yet general techniques for addressing some of these challenges.
Revised: We believe this work to be a step towards simple yet general techniques for addressing some of these challenges.

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We will use the shorthand || · || to denote the L 2 -norm and ∇L = ∇ θ L , where θ is the parameter vector.
Revised: We will use the shorthand || · || to denote the L 2 -norm and ∇L = ∇ θ L , where θ is the parameter vector.

Original: Let g 1 =
Revised: Let g 1 =

Original: ∇L 1 , g 2 = ∇L 2 , and φ be the angle between g 1 and g 2 .
Revised: ∇L 1 , g 2 = ∇L 2 , and φ be the angle between g 1 and g 2 .

Original: At each PCGrad update, we have two cases: cos ( φ ) ≥ 0 or cos( φ < 0) .
Revised: At each PCGrad update, we have two cases: cos ( φ ) ≥ 0 or cos( φ < 0) .

Original: If cos( φ ) ≥ 0 , then we apply the standard gradient descent update using t ≤ 1 L , which leads to a strict decrease in the objective function value L ( φ ) unless ∇L ( φ ) = 0 , which occurs only when θ = θ ∗ (Boyd & Vandenberghe, 2004).
Revised: If cos( φ ) ≥ 0 , then we apply the standard gradient descent update using t ≤ 1 L , which leads to a strict decrease in the objective function value L ( φ ) unless ∇L ( φ ) = 0 , which occurs only when θ = θ ∗ (Boyd & Vandenberghe, 2004).

Original: In the case that cos( φ ) < 0 , we proceed as follows:
Revised: In the case that cos( φ ) < 0 , we proceed as follows:

Original: Our assumption that ∇L is Lipschitz continuous with constant L implies that ∇ 2 L ( θ ) − LI is a negative semideﬁnite matrix.
Revised: Our assumption that ∇L is Lipschitz continuous with constant L implies that ∇ 2 L ( θ ) − LI is a negative semideﬁnite matrix.

Original: Using this fact, we can perform a quadratic expansion of L around L ( θ ) and obtain the following inequality:
Revised: Using this fact, we can perform a quadratic expansion of L around L ( θ ) and obtain the following inequality:

Original: [Equation]
Revised: [Equation]

Original: Now, we can plug in the PCGrad update by letting θ
Revised: Now, we can plug in the PCGrad update by letting θ

Original: + = θ − t ( ∇L ( θ ) −
Revised: + = θ − t ( ∇L ( θ ) −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 |
Revised: 2 |

Original: [Equation]
Revised: [Equation]

Original: 2 g 1 −
Revised: 2 g 1 −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 || g
Revised: 2 || g

Original: || 2 g 2 ) .
Revised: || 2 g 2 ) .

Original: We then get:
Revised: We then get:

Original: [Equation]
Revised: [Equation]

Original: (Expanding, using the identity ∇L ( θ ) =
Revised: (Expanding, using the identity ∇L ( θ ) =

Original: g 1 + g 2 )
Revised: g 1 + g 2 )

Original: [Equation]
Revised: [Equation]

Original: (Expanding further and re-arranging terms)
Revised: (Expanding further and re-arranging terms)

Original: [Equation]
Revised: [Equation]

Original: (Using the identity cos( φ ) =
Revised: (Using the identity cos( φ ) =

Original: [Equation]
Revised: [Equation]

Original: (Note that cos( φ ) < 

Revised: (Note that cos( φ ) <

Original: so the ﬁnal term is non-negative)
Revised: so the ﬁnal term is non-negative)

Original: Using t ≤ 1 L , we know that − (1 − 12 Lt ) = 12 Lt − 1 ≤ 12 L (1 /L ) − 1 = − 12 and Lt 2 ≤ t .
Revised: Using t ≤ 1 L , we know that − (1 − 12 Lt ) = 12 Lt − 1 ≤ 12 L (1 /L ) − 1 = − 12 and Lt 2 ≤ t .

Original: Plugging this into the last expression above, we can conclude the following:
Revised: Plugging this into the last expression above, we can conclude the following:

Original: [Equation]
Revised: [Equation]

Original: If cos( φ ) > − 1 , then 12 t (1 − cos 2 ( φ )) (cid:107)∇L ( θ ) (cid:107) 2 will always be positive unless ∇L ( θ ) = 0 .
Revised: If cos( φ ) > − 1 , then 12 t (1 − cos 2 ( φ )) (cid:107)∇L ( θ ) (cid:107) 2 will always be positive unless ∇L ( θ ) = 0 .

Original: This inequality implies that the objective function value strictly decreases with each iteration where cos( φ ) > − 1 .
Revised: This inequality implies that the objective function value strictly decreases with each iteration where cos( φ ) > − 1 .

Original: Hence repeatedly applying PCGrad process can either reach the optimal value L ( θ )
Revised: Hence repeatedly applying PCGrad process can either reach the optimal value L ( θ )

Original: =
Revised: =

Original: L ( θ ∗ ) or cos( φ ) = − 1 , in which case 12 t (1 − cos 2 ( φ ))
Revised: L ( θ ∗ ) or cos( φ ) = − 1 , in which case 12 t (1 − cos 2 ( φ ))

Original: (cid:107)∇L ( θ ) (cid:107) 2 = 0 .
Revised: (cid:107)∇L ( θ ) (cid:107) 2 = 0 .

Original: Note that this result only holds when we choose t to be small enough, i.e. t ≤ 1 L .
Revised: Note that this result only holds when we choose t to be small enough, i.e. t ≤ 1 L .

Original: B 2D O PTIMIZATION L ANDSCAPE D ETAILS
Revised: B 2D O PTIMIZATION L ANDSCAPE D ETAILS

Original: To produce the 2D optimization visualizations in Figure 1, we used a parameter vector θ =
Revised: To produce the 2D optimization visualizations in Figure 1, we used a parameter vector θ =

Original: [ θ 1 , θ 2 ] ∈ Rand the following task loss functions:
Revised: [ θ 1 , θ 2 ] ∈ Rand the following task loss functions:

Original: [Equation]
Revised: [Equation]

Original: The multi-task objective is L ( θ ) =
Revised: The multi-task objective is L ( θ ) =

Original: L 1 ( θ ) + L 2 ( θ ) .
Revised: L 1 ( θ ) + L 2 ( θ ) .

Original: We initialized θ =
Revised: We initialized θ =

Original: [0 . 5 , − 3] and performed 500,000 gradient updates to minimize L using the Adam optimizer with learning rate 0 .
Revised: [0 . 5 , − 3] and performed 500,000 gradient updates to minimize L using the Adam optimizer with learning rate 0 .

Original: We compared using Adam for each update to using Adam in conjunction with the PCGrad method presented in Section 3.2.
Revised: We compared using Adam for each update to using Adam in conjunction with the PCGrad method presented in Section 3.2.

Original: C E XPERIMENTAL R ESULTS ON M ULTI MNIST
Revised: C E XPERIMENTAL R ESULTS ON M ULTI MNIST

Original: Following the same set-up in Sener & Koltun (2018), for each image, we sample a different one uniformly at random.
Revised: Following the same set-up in Sener & Koltun (2018), for each image, we sample a different one uniformly at random.

Original: Then we put one of the image on the top left and the other one on the bottom right.
Revised: Then we put one of the image on the top left and the other one on the bottom right.

Original: The two tasks in the multi-task learning problem are to classify the digits on the top left (task-L) and bottom right (task-R) respectively.
Revised: The two tasks in the multi-task learning problem are to classify the digits on the top left (task-L) and bottom right (task-R) respectively.

Original: We construct such 60 K examples.
Revised: We construct such 60 K examples.

Original: We combine PCGrad with the same backbone architecture used in (Sener & Koltun, 2018) and compare its performance to Sener & Koltun (2018) by running the open-sourced code provided in (Sener & Koltun, 2018).
Revised: We combine PCGrad with the same backbone architecture used in (Sener & Koltun, 2018) and compare its performance to Sener & Koltun (2018) by running the open-sourced code provided in (Sener & Koltun, 2018).

Original: As shown in
Revised: As shown in

Original: Table 3, our method results 0.13% and 0.55% improvement over Sener & Koltun (2018) in left and right digit accuracy respectively.
Revised: Table 3, our method results 0.13% and 0.55% improvement over Sener & Koltun (2018) in left and right digit accuracy respectively.

Original: [Table]
Revised: [Table]

Original: D A BLATION STUDY ON THE TASK ORDER
Revised: D A BLATION STUDY ON THE TASK ORDER

Original: As stated on line 4 in Algorithm 1, we sample the tasks from the batch and randomly shufﬂe the order of the tasks before performing the update steps in PCGrad.
Revised: As stated on line 4 in Algorithm 1, we sample the tasks from the batch and randomly shufﬂe the order of the tasks before performing the update steps in PCGrad.

Original: With random shufﬂing, we make
Revised: With random shufﬂing, we make

Original: PCGrad symmetric w.r.t.
Revised: PCGrad symmetric w.r.t.

Original: the task order in expectation.
Revised: the task order in expectation.

Original: In Figure 7, we observe that PCGrad with a random task order achieves better performance between PCGrad with a ﬁxed task order in the setting of MT50 where the number of tasks is large and the conﬂicting gradient phenomenon is much more likely to happen.
Revised: In Figure 7, we observe that PCGrad with a random task order achieves better performance between PCGrad with a ﬁxed task order in the setting of MT50 where the number of tasks is large and the conﬂicting gradient phenomenon is much more likely to happen.

Original: E E XPERIMENT D ETAILS E. D ETAILED E XPERIMENT S ET - UP
Revised: E E XPERIMENT D ETAILS E. D ETAILED E XPERIMENT S ET - UP

Original: For our CIFAR-100 multi-task experiment, we adopt the architecture used in Rosenbaum et al.
Revised: For our CIFAR-100 multi-task experiment, we adopt the architecture used in Rosenbaum et al.

Original: (2019), which is a convolutional neural network that consists of 3 convolutional layers with 160 3 × 3 ﬁlters each layer and 2 fully connected layers with 320 hidden units.
Revised: (2019), which is a convolutional neural network that consists of 3 convolutional layers with 160 3 × 3 ﬁlters each layer and 2 fully connected layers with 320 hidden units.

Original: As for experiments on the NYUv dataset, we follow Liu et al.
Revised: As for experiments on the NYUv dataset, we follow Liu et al.

Original: (2018) to use SegNet (Badrinarayanan et al., 2017) as the backbone architecture.
Revised: (2018) to use SegNet (Badrinarayanan et al., 2017) as the backbone architecture.

Original: Our reinforcement learning experiments all use the SAC (Haarnoja et al., 2018)
Revised: Our reinforcement learning experiments all use the SAC (Haarnoja et al., 2018)

Original: algorithm as the base algorithm, where the actor and the critic are represented as 6-layer fully-connected feedforward neural networks for all methods.
Revised: algorithm as the base algorithm, where the actor and the critic are represented as 6-layer fully-connected feedforward neural networks for all methods.

Original: The numbers of hidden units of each layer of the neural networks are 160 , 300 and 200 for MT10, MT50 and goal-conditioned RL respectively.
Revised: The numbers of hidden units of each layer of the neural networks are 160 , 300 and 200 for MT10, MT50 and goal-conditioned RL respectively.

Original: We use ﬁve algorithms as baselines in the CIFAR-100 multi-task experiment: task speciﬁc-1-fc (Rosenbaum et al., 2018): a convolutional neural network shared across tasks except that each task has a separate last fully-connected layer, task speciﬁc-1-fc (Rosenbaum et al., 2018) : all the convolutional layers shared across tasks with separate fully-connected layers for each task, cross stitch-all-fc (Misra et al., 2016b): one convolutional neural network per task along with cross-stitch units to share features across tasks, routing-all-fc + WPL (Rosenbaum et al., 2019): a network that employs a trainable router trained with multi-agent RL algorithm (WPL) to select trainable functions for each task, independent : training separate neural networks for each task, independent + natural gradient : training separate neural networks for each task using the K-FAC method to approximate the natural gradient.
Revised: We use ﬁve algorithms as baselines in the CIFAR-100 multi-task experiment: task speciﬁc-1-fc (Rosenbaum et al., 2018): a convolutional neural network shared across tasks except that each task has a separate last fully-connected layer, task speciﬁc-1-fc (Rosenbaum et al., 2018) : all the convolutional layers shared across tasks with separate fully-connected layers for each task, cross stitch-all-fc (Misra et al., 2016b): one convolutional neural network per task along with cross-stitch units to share features across tasks, routing-all-fc + WPL (Rosenbaum et al., 2019): a network that employs a trainable router trained with multi-agent RL algorithm (WPL) to select trainable functions for each task, independent : training separate neural networks for each task.

Original: For comparisons on the NYUv2 dataset, we consider 5 baselines: Single Task, One Task : the vanilla
Revised: For comparisons on the NYUv2 dataset, we consider 5 baselines: Single Task, One Task : the vanilla

Original: SegNet used for single-task training, Single Task, STAN (Liu et al., 2018): the single-task version of MTAN as mentioned below, Multi-Task, Split, Wide / Deep
Revised: SegNet used for single-task training, Single Task, STAN (Liu et al., 2018): the single-task version of MTAN as mentioned below, Multi-Task, Split, Wide / Deep

Original: (Liu et al., 2018): the standard
Revised: (Liu et al., 2018): the standard

Original: SegNet shared for all three tasks except that each task has a separate last layer for ﬁnal task-speciﬁc prediction with two variants Wide and Deep speciﬁed in Liu et al.
Revised: SegNet shared for all three tasks except that each task has a separate last layer for ﬁnal task-speciﬁc prediction with two variants Wide and Deep speciﬁed in Liu et al.

Original: (2018), Multi-Task Dense : a shared network followed by separate task-speciﬁc networks, Multi-Task Cross-Stitch (Misra et al., 2016b): similar to the baseline used in CIFAR-100 experiment but with SegNet as the backbone,
Revised: (2018), Multi-Task Dense : a shared network followed by separate task-speciﬁc networks, Multi-Task Cross-Stitch (Misra et al., 2016b): similar to the baseline used in CIFAR-100 experiment but with SegNet as the backbone,

Original: MTAN (Liu et al., 2018): a shared network with a soft-attention module for each task.
Revised: MTAN (Liu et al., 2018): a shared network with a soft-attention module for each task.

Original: On the multi-task and goal-conditioned RL domain, we apply PCGrad to the vanilla SAC algorithm with task encoding as part of the input to the actor and the critic as described in Section 4 and compare our method to the vanilla SAC without PCGrad and training actors and critics for each task individually ( Independent ).
Revised: On the multi-task and goal-conditioned RL domain, we apply PCGrad to the vanilla SAC algorithm with task encoding as part of the input to the actor and the critic as described in Section 4 and compare our method to the vanilla SAC without PCGrad and training actors and critics for each task individually ( Independent ).

Original: E. G OAL - CONDITIONED E XPERIMENT D ETAILS
Revised: E. G OAL - CONDITIONED E XPERIMENT D ETAILS

Original: We use the pushing environment from the Meta-World benchmark (Yu et al., 2019) as shown in
Revised: We use the pushing environment from the Meta-World benchmark (Yu et al., 2019) as shown in

Original: Figure 3.
Revised: Figure 3.

Original: In this environment, the table spans from [ − 0 . 4 , 0 . 2] to [0 . 4 , 1 . 0] in the 2D space.
Revised: In this environment, the table spans from [ − 0 . 4 , 0 . 2] to [0 . 4 , 1 . 0] in the 2D space.

Original: To construct the goals, we sample the intial positions of the puck from the range [ − 0 .
Revised: To construct the goals, we sample the intial positions of the puck from the range [ − 0 .

Original: 2 , 0 . 6] to [0 . 2 , 0 . 7] on the table and the goal positions from the range [ − 0 .
Revised: 2 , 0 . 6] to [0 . 2 , 0 . 7] on the table and the goal positions from the range [ − 0 .

Original: 2 , 0 . 85] to [0 . 2 , 0 . 95] on the table.
Revised: 2 , 0 . 85] to [0 . 2 , 0 . 95] on the table.

Original: The goal is represented as a concatenation of the initial puck position and the goal position.
Revised: The goal is represented as a concatenation of the initial puck position and the goal position.

Original: Since in the goal-conditioned setting, the task distribution is continuous, we sample a minibatch of 9 goals and 128 samples per goal at each training iteration and also sample 600 samples per goal in the minibatch at each data collection step.
Revised: Since in the goal-conditioned setting, the task distribution is continuous, we sample a minibatch of 9 goals and 128 samples per goal at each training iteration and also sample 600 samples per goal in the minibatch at each data collection step.

Original: E. F ULL NYU V 2 R ESULTS
Revised: E. F ULL NYU V 2 R ESULTS

Original: We provide the full comparison on the NYUv2 dataset in Table
Revised: We provide the full comparison on the NYUv2 dataset in Table

Original: [Table]
Revised: [Table]

Original: [Table]