R OTO G RAD : G RADIENT H OMOGENIZATION IN M ULTITASK L EARNING
A BSTRACT
Multitask learning is being increasingly adopted in applications domains like computer vision and reinforcement learning.
However, optimally exploiting its advantages remains a major challenge due to the effect of negative transfer.
Previous works have tracked down this issue to the disparities in gradient magnitudes and directions across tasks when optimizing the shared network parameters.
While recent work has acknowledged that negative transfer is a two-fold problem, existing approaches fall short.
These methods only focus on either homogenizing the gradient magnitude across tasks; or greedily change the gradient directions, overlooking future conﬂicts.
In this work, we introduce RotoGrad, an algorithm that tackles negative transfer as a whole: it jointly homogenizes gradient magnitudes and directions, while ensuring training convergence.
We show that RotoGrad outperforms competing methods in complex problems, including multi-label classiﬁcation in CelebA and computer vision tasks in the NYUv2 dataset.
A Pytorch implementation can be found in https://github.com/adrianjav/rotograd .
I NTRODUCTION
As neural network architectures get larger in order to solve increasingly more complex tasks, the idea of jointly learning multiple tasks (for example, depth estimation and semantic segmentation in computer vision) with a single network is becoming more and more appealing.
This is precisely the idea of multitask learning (MTL) (Caruana, 1993), which promises higher performance in the individual tasks and better generalization to unseen data, while drastically reducing the number of parameters (Ruder, 2017).
Unfortunately, sharing parameters between tasks may also lead to difﬁculties during training as tasks compete for shared resources, often resulting in poorer results than solving individual tasks, a phenomenon known as negative transfer (Ruder, 2017).
Previous works have tracked down this issue to the two types of differences between task gradients.
First, differences in magnitude across tasks can make some tasks dominate the others during the learning process.
Several methods have been proposed to homogenize gradient magnitudes such as MGDA-UB (Sener & Koltun, 2018),
GradNorm (Chen et al., 2018), or IMTL-G Liu et al.
(2021b).
However, little attention has been put towards the second source of the problem: conﬂicting directions of the gradients for different tasks.
Due to the way gradients are added up, gradients of different tasks may cancel each other out if they point to opposite directions of the parameter space, thus leading to a poor update direction for a subset or even all tasks.
Only very recently a handful of works have started to propose methods to mitigate the conﬂicting gradients problem, for example, by removing conﬂicting parts of the gradients (Yu et al., 2020), or randomly ‘dropping’ some elements of the gradient vector (Chen et al., 2020).
In this work we propose RotoGrad, an algorithm that tackles negative transfer as a whole by homogenizing both gradient magnitudes and directions across tasks.
RotoGrad addresses the gradient magnitude discrepancies by re-weighting task gradients at each step of the learning, while encouraging learning those tasks that have converged the least thus far.
In that way, it makes sure that no task is overlooked during training.
Additionally, instead of directly modifying gradient directions, RotoGrad smoothly rotates the shared feature space differently for each task, seamlessly aligning gradients in t
V a n ill a
R o t
o G r a d
[Figure]
the long run. As shown by our theoretical insights, the cooperation between gradient magnitudeand direction-homogenization ensures the stability of the overall learning process. Finally, we run extensive experiments to empirically demonstrate that RotoGrad leads to stable (convergent) learning, scales up to complex network architectures, and outperforms competing methods in multi-label classiﬁcation settings in CIFAR10 and CelebA, as well as in computer vision tasks using the NYUv dataset.
Moreover, we provide a simple-to-use library to include RotoGrad in any Pytorch pipeline.
M ULTITASK LEARNING AND NEGATIVE TRANSFER
The goal of MTL is to simultaneously learn K different tasks, that is, ﬁnding K mappings from a common input dataset X ∈ R N × D to a task-speciﬁc set of labels Y k ∈ Y Nk .
Most settings consider a hard-parameter sharing architecture, which is characterized by two components: the backbone and heads networks.
The backbone uses a set of shared parameters, θ , to transform each input x ∈ X into a shared intermediate representation z
= f ( x ; θ ) ∈ R d , where d is the dimensionality of z .
Additionally, each task k
= 1 , 2 , . .
, K has a head network h k , with exclusive parameters φ k , that takes this intermediate feature z and outputs the prediction h k ( x ) =
h k ( z ; φ k ) for the corresponding task.
This architecture is illustrated in Figure 2, where we have added task-speciﬁc rotation matrices
R k that will be necessary for the proposed approach, RotoGrad.
Note that the general architecture described above is equivalent to the one in Figure 2 when all rotations R k correspond to identity matrices, such that r k = z for all k .
[Equation]
MTL aims to learn the architecture parameters θ , φ 1 , φ 2 , . .
. , φ K by simultaneously minimizing all task losses, that is, L k ( h k ( x ) , y k ) for k = 1 , . . .
, K .
Although this is a priori a multiobjective optimization problem (Sener & Koltun, 2018), in practice a single surrogate loss consisting of a linear combination of the task losses,
L = (cid:80)
k ω k L k , is optimized.
While this approach leads to a simpler optimization problem, it may also trigger negative transfer between tasks, hurting the overall MTL performance due to an imbalanced competition among tasks for the shared parameters (Ruder, 2017).
The negative transfer problem can be studied through the updates of the shared parameters θ .
At each training step, θ is updated according to a linear combination of task gradients, ∇ θ L = (cid:80) k ω k ∇ θ L k , which may suffer from two problems.
First, magnitude differences of the gradients across tasks may lead to a subset of tasks dominating the total gradient, and therefore to the model prioritizing them over the others.
Second, conﬂicting directions of the gradients across tasks may lead to update directions that do not improve any of the tasks.
Figure 1 shows an example of poor direction updates
In this work, we tackle negative transfer as a whole by homogenizing tasks gradients both in magnitude and direction.
To reduce overhead, we adopt the usual practice and homogenize gradients with respect to the shared feature z (rather than θ ), as all tasks share gradient up to that point, ∇ θ L k = ∇ θ z · ∇ z L k .
Thus, from now on we focus on feature-level task gradients ∇ z L k .
R OTO G RAD
In this section we introduce RotoGrad, a novel algorithm that addresses the negative transfer problem as a whole.
RotoGrad consists of two building blocks which, respectively, homogenize task-gradient magnitudes and directions.
Moreover, these blocks complement each other and provide convergence guarantees of the network training.
Next, we detail each of these building blocks and show how they are combined towards an effective MTL learning process.
3. G RADIENT - MAGNITUDE HOMOGENIZATION
As discussed in §2, we aim to homogenize gradient magnitudes across tasks, as large magnitude disparities can lead to a subset of tasks dominating the learning process.
Thus, the ﬁrst goal of
RotoGrad is to homogenize the magnitude of the gradients across tasks at each step of the training.
Let us denote the feature-level task gradient of the k -th task for the n -th datapoint, at iteration t , by g n,k := ∇ z L k ( h k ( x n ) , y n,k ) , and its batch versions by G (cid:62) k :=
[ g 1 ,k , g 2 ,k , . . . , g B,k ] , where B is the batch size.
Then, equalizing gradient magnitudes amounts to ﬁnding weights ω
k
that normalize and scale each gradient G k , that is,
[Equation]
where U k :
=
G k || G k || denotes the normalized task gradient and C is the target magnitude for all tasks.
Note that, in the above expression, C is a free parameter that we need to select.
In RotoGrad, we select C such that all tasks converge at a similar rate.
We motivate this choice by the fact that, by scaling all gradients, we change their individual step size, interfering with the convergence guarantees provided by their Lipschitz-smoothness (for an introduction to optimization see, for example, (Nesterov, 2004)).
Therefore, we seek for the value of C providing the best step-size for those tasks that have converged the least up to iteration t .
Speciﬁcally, we set C to be a convex combination of the task-wise gradient magnitudes, C :
=
(cid:80) k
α k
|| G k || , where the weights
α 1 , α 2 , . .
. , α K measure the relative convergence of each task and sum up to one, that is,
[Equation]
with G 0 k being the initial gradient of the k -th task, i.e., the gradient at iteration t = 0 of the training.
As a result, we obtain a (hyper)parameter-free approach that equalizes the gradient magnitude across tasks to encourage learning slow-converging tasks.
Note that the resulting approach resembles
Normalized Gradient Descent (NGD) (Cortés, 2006) for single-task learning, which has been proved to quickly escape saddle points during optimization (Murray et al., 2019).
Thus, we expect a similar behavior for RotoGrad, where slow-converging tasks will force quick-converging tasks to escape from saddle points.
Whilst the algorithm works well in general, its simplicity also facilitates unfavorable settings.
For example, in the presence of noisy tasks that do not progress; or in scenarios where, when one task improves, there is always another task that deteriorates.
In Appendix A we show that, when gradients do not conﬂict in direction with each other (which we pursue next), following the gradient C (cid:80) k
U k improves all task losses for the given batch.
This result, while simple, provides insights in favor of having as desideratum of an efﬁcient MTL pipeline the absence of conﬂicting gradients.
3. G RADIENT - DIRECTION HOMOGENIZATION
In the previous subsection, we have shown that avoiding conﬂicting gradients may not only be necessary to avoid negative transfer, but also to ensure the stability of the training.
In this section we introduce the second building block of RotoGrad, an algorithm that homogenizes task-gradient directions.
The main idea of this approach is to smoothly rotate the feature-space z in order to reduce the gradient conﬂict between tasks—in following iterations—of the training by bringing (local) optima for different tasks closer to each other (in the parameter space).
As a result, it complements the previous magnitude-scaling approach and reduces the likelihood of the training to diverge.
In order to homogenize gradients, for each task k
= 1 , . .
, K , RotoGrad introduces a matrix
R k
so that, instead of optimizing L k ( z ) with z being the last shared representation, we optimize an equivalent (in optimization terms, as it is a bijective mapping) loss function L k ( R k z ) .
As we are only interested in changing directions (not the gradient magnitudes), we choose R k ∈ SO ( d ) to be a rotation matrix 1 leading to per-task representations r k :=
R k z .
RotoGrad thus extends the standard
MTL architecture by adding task-speciﬁc rotations before each head, as depicted in Figure 2.
Unlike all other network parameters, matrices R k do not seek to reduce their task’s loss.
Instead, these additional parameters are optimized to reduce the direction conﬂict of the gradients across tasks.
To this end, for each task we optimize R k to maximize the batch-wise cosine similarity or, equivalently, to minimize
[Equation]
where (cid:101) g n,k := ∇ r k L k ( h k ( x n ) , y n,k )) (which holds that g n,k = R (cid:62) k (cid:101) g n,k ) and v n is the target vector that we want all task gradients pointing towards.
We set the target vector v n to be the gradient we would have followed if all task gradients weighted the same, that is, v n := 1 K (cid:80) k u n,k , where u n,k is a row vector of the normalized batch gradient matrix
U k , as deﬁned before.
As a result, in each training step of RotoGrad we simultaneously optimize the following two problems:
[Equation]
The above problem can be interpreted as a Stackelberg game: a two player-game in which leader and follower alternately make moves in order to minimize their respective losses, L l and L f , and the leader knows what will be the follower’s response to their moves.
Such an interpretation allows us to derive simple guidelines to guarantee training convergence—that is, that the network loss does not oscillate as a result of optimizing the two different objectives in Equation 4.
Speciﬁcally, following
Fiez et al.
(2020), we can ensure that problem 4 converges as long as the rotations’ optimizer (leader) is a slow-learner compared with the network optimizer (follower).
That is, as long as we make the rotations’ learning rate decrease faster than that of the network, we know that RotoGrad will converge to a local optimum for both objectives.
A more extensive discussion can be found in Appendix B.
3. R OTO G RAD : THE FULL PICTURE
After the two main building blocks of RotoGrad, we can now summarize the overall proposed approach in Algorithm 1.
At each step, RotoGrad ﬁrst homogenizes the gradient magnitudes such that there is no dominant task and the step size is set by the slow-converging tasks.
Additionally,
RotoGrad smoothly updates the rotation matrices—using the local information given by the task gradients—to seamlessly align task gradients in the following steps, thus reducing direction conﬂicts.
3. P RACTICAL CONSIDERATIONS
In this section, we discuss the main practical considerations to account for when implementing
RotoGrad and propose efﬁcient solutions.
Unconstrained optimization.
As previously discussed, parameters R k are deﬁned as rotation matrices, and thus the Rotation optimization in problem 4 is a constrained problem.
While this would typically imply using expensive algorithms like Riemannian gradient descent (Absil et al., 2008), we can leverage recent work on manifold parametrization (Casado & Martínez-Rubio, 2019) and, instead, apply unconstrained optimization methods by automatically 2 parametrizing R k via exponential maps on the Lie algebra of SO ( d ) . Input input samples Output backbone heads) parameters θ { R
1: compute shared feature Z = f ( X ; θ ) 2: for k = 1 , 2 , . . , K do 3: compute task-speciﬁc loss L k = (cid:80) n L k ( h k ( R k z n ; φ k ) , y n,k ) 4: compute gradient of shared feature G k = ∇ z L k 5: compute gradient of task-speciﬁc feature (cid:101)
G k = R k G k (cid:46) Treated as constant w.r.t.
R k . 6: compute unitary gradients U = G k / ||
G k || 7: compute relative task convergence α k = || G k || /
||
G 0 k || 8: end for 9: make { α k } sum up to one [ α 1 , α 2 , . . . , α K ] = [ α 1 , α 2 , . . . , α K ] / (cid:80) k
α k 10: compute shared magnitude C = (cid:80) k
α k || G k || 11: update backbone parameters θ
= θ − ηC (cid:80) k
U k 12: compute target vector V = 1 K (cid:80) k
U k 13: for k = 1 , 2 , . .
, K do 14: compute RotoGrad’s loss L roto k
= − (cid:80) n (cid:104) R (cid:62) k (cid:101) g n,k , v n (cid:105) 15: update RotoGrad’s parameters R k = R k − η roto ∇ R k L roto k 16: update head’s parameters φ k = φ k − η ∇ φ k L k 17: end for
Memory efﬁciency and time complexity.
As we need one rotation matrix per task, we have to store O
( Kd 2 ) additional parameters.
In practice, we only need Kd ( d − 1) / 2 parameters due to the aforementioned parametrization and, in most cases, this amounts to a small part of the total number of parameters.
Moreover, as described by Casado & Martínez-Rubio (2019), parametrizing R k enables efﬁcient computations compared with traditional methods, with a time complexity of O ( d 3 ) independently of the batch size.
In our case, the time complexity is of O ( Kd 3 ) , which scales better with respect to the number of tasks than existing methods (for example, O ( K 2 d ) for PCGrad (Yu et al., 2020)).
Forward-pass caching and GPU parallelization can further reduce training time.
Scaling-up RotoGrad.
Despite being able to efﬁciently compute and optimize the rotation matrix
R k , in domains like computer vision, where the size d of the shared representation z is large, the time complexity for updating the rotation matrix may become comparable to the one of the network updates.
In those cases, we propose to only rotate a subspace of the feature space, that is, rotate only m << d dimensions of z .
Then, we can simply apply a transformation of the form r k =
[ R k z 1: m , z m
+1: d ] , where z a : b denotes the elements of z with indexes a, a + 1 , . . .
, b .
While there exist other possible solutions, such as using block-diagonal rotation matrices R k , we defer them to future work.
I LLUSTRATIVE EXAMPLES
In this section, we illustrate the behavior of RotoGrad in two synthetic scenarios, providing clean qualitative results about its effect on the optimization process.
Appendix C.1 provides a detailed description of the experimental setups.
To this end, we propose two different multitask regression problems of the form
[Equation]
where ϕ is a test function with a single global optimum whose position is parametrized by the second argument, that is, both tasks are identical (and thus related) up to a translation.
We use a single input x ∈ R 2 and drop task-speciﬁc network parameters.
As backbone, we take a simple network of the form z
= W 2 max( W 1 x + b 1 , 0)
+ b 2 with b 1 ∈ R 10 , b 2 ∈ R 2 , and W 1 , W (cid:62) 2 ∈ R 10 × 2 .
For the ﬁrst experiment we choose a simple (avocado-shaped) convex objective function and, for the second one, we opt for a non-convex function with several local optima and a single global optimum.
Figure 1 shows the training trajectories in the presence (and absence) of RotoGrad in both experiments, depicted as level plots in the space of z and r k , respectively.
We can observe that in the ﬁrst experiment (Figure 1a), RotoGrad ﬁnds both optima by rotating the feature space and matching the (unique) local optima of the tasks.
Similarly, the second experiment (Figure 1b) shows that, as we have two symmetric tasks and a non-equidistant starting point, in the vanilla case the optimization is dominated by the task with an optimum closest to the starting point.
RotoGrad avoids this behavior by equalizing gradients and, by aligning gradients, is able to ﬁnd the optima of both functions.
R ELATED W ORK
Understanding and improving the interaction between tasks is one of the most fundamental problems of MTL, since any improvement in this regard would translate to all MTL systems.
Consequently, several approaches to address this problem have been adopted in the literature.
Among the different lines of work, the one most related to the present work is gradient homogenization.
Gradient homogenization.
Since the problem is two-fold, there are two main lines of work.
On the one hand, we have task-weighting approaches that focus on alleviating magnitude differences.
Similar to us, GradNorm (Chen et al., 2018) attempts to learn all tasks at a similar rate, yet they propose to learn these weights as parameters.
Instead, we provide a closed-form solution in Equation 1, and so does IMTL-G Liu et al.
(2021b).
However, IMTL-G scales all task gradients such that all projections of G onto G k are equal.
MGDA-UB (Sener & Koltun, 2018), instead, adopts an iterative method based on the Frank-Wolfe algorithm in order to ﬁnd the set of weights { ω k } (with (cid:80)
k ω k
= 1 ) such that (cid:80) k ω
k G k has minimum norm.
On the other hand, recent works have started to put attention on the conﬂicting direction problem.
Maninis et al.
and Sinha et al.
(2018) proposed to make task gradients statistically indistinguishable via adversarial training.
More recently, PCGrad (Yu et al., 2020) proposed to drop the projection of one task gradient onto another if they are in conﬂict, whereas GradDrop (Chen et al., 2020) randomly drops elements of the task gradients based on a sign-purity score.
Contemporaneously to this work, improved versions of MGDA (Désidéri, 2012) and PCGrad have been proposed by Liu et al.
(2021a)
and Wang et al.
(2021), respectively.
The literature also includes approaches which, while orthogonal to the gradient homogenization, are complementary to our work and thus could be used along with RotoGrad.
Next, we provide a brief overview of them.
A prominent approach for MTL is task clustering, that is, selecting which tasks should be learned together.
This approach dates back to the original task-clustering algorithm (Thrun & O’Sullivan, 1996), but new work in this direction keeps coming out (Standley et al., 2020; Zamir et al., 2018;
Shen et al., 2021; Fifty et al., 2021).
Alternative approaches, for example, scale the loss of each task differently based on different criteria such as task uncertainty (Kendall et al., 2018), task prioritization (Guo et al., 2018), or similar loss magnitudes (Liu et al., 2021b).
Moreover, while most models fall into the hard-parameter sharing umbrella, there exists other architectures in the literature.
Soft-parameter sharing architectures (Ruder, 2017), for example, do not have shared parameters but instead impose some kind of shared restrictions to the entire set of parameters.
An interesting approach consists in letting the model itself learn which parts of the architecture should be used for each of the tasks (Guo et al., 2020; Misra et al., 2016; Sun et al., 2020; Vandenhende et al., 2020).
Other architectures, such as MTAN (Liu et al., 2019b), make use of task-speciﬁc attention to select relevant features for each task.
Finally, similar issues have also been studied in other domains like meta-learning (Flennerhag et al., 2019) and continual learning (Lopez-Paz & Ranzato, 2017).
E XPERIMENTS
In this section we assess the performance of RotoGrad on a wide range of datasets and MTL architectures.
First, we check the effect of the learning rates of the rotation and network updates on the stability of RotoGrad.
Then, with the goal of applying RotoGrad to scenarios with large sizes of z , we explore the effect of rotating only a subspace of z .
Finally, we compare our approach with competing MTL solutions in the literature, showing that RotoGrad consistently outperforms all existing methods.
Refer to Appendix C for a more details on the experiments and additional results.
Relative task improvement.
Since MTL uses different metrics for different tasks, throughout this section we group results by means of the relative task improvement, ﬁrst introduced by Maninis et al.
Given a task k , and the metrics obtained during test time by a model, M k , and by a baseline model, S k , which consists of K networks trained on each task individually, the relative task improvement for the k -th task is deﬁned as
[Equation]
where l k = 1 if M k < S k means that our model performs better than the baseline in the k -th task, and l k = 0 otherwise.
We depict our results using different statistics of ∆ k such as its mean ( avg k ∆ k ), maximum ( max k ∆ k ), and median ( med k ∆ k ) across tasks.
Statistical signiﬁcance.
We highlight signiﬁcant improvements according to a one-sided paired t-test ( α = 0 . 05 ), with respect to MTL with vanilla optimization (marked with † in each table).
6. T RAINING STABILITY
At the end of §3.2 we discussed that, by casting problemas a Stackelberg game, we have convergence guarantees when the rotation optimizer is the slow-learner.
Next, we empirically show this necessary condition.
Experimental setup.
Similar to (Sener & Koltun, 2018), we use a multitask version of MNIST (LeCun et al., 2010) where each image is composed of a left and right digit, and use as backbone a reduced version of LeNet (LeCun et al., 1998) with light-weight heads.
Besides the leftand right-digit classiﬁcation proposed in (Sener & Koltun, 2018), we consider three other quantities to predict: i) sum of digits; ii) parity of the digit product; and iii) number of active pixels.
The idea here is to enforce all digit-related tasks to cooperate (positive transfer), while the (orthogonal) image-related task should not disrupt these learning dynamics.
Results.
Figure 3 shows the effect—averaged over ten independent runs—of changing the rotations’ learning rate in terms of test error in the sum task, while the rest of tasks are shown in Appendix C.2.
We observe that, the bigger the learning rate is, in comparison to that of the network’s parameters ( 1 e − 3 ), the higher and noisier the test error becomes.
MSE keeps decreasing as we lower the learning rate, reaching a sweet-spot at half the network’s learning rate ( 5 e − 4 ).
For smaller values, the rotations’ learning is too slow and results start to resemble those of the vanilla case, in which no rotations are applied (leftmost box in Figure 3).
6. R OTO G RAD BUILDING BLOCKS
In this section, we empirically evaluate to which extent each of the RotoGrad building blocks, that we denote Scale Only (§3.1) and Rotate Only (§3.2), contribute to the performance gain in MTL.
Experimental setup.
We test all methods on three different tasks of NYUv2 (Couprie et al., 2013): 13-class semantic segmentation; depth estimation; and normal surfaces estimation.
To speed up training, all images were resized to 288 × 384 resolution; and data augmentation was applied to alleviate overﬁtting.
As MTL architecture, we use SegNet (Badrinarayanan et al., 2017) where the decoder is split into three convolutional heads.
This is the same setup as that of Liu et al.
(2019b).
Results.
The three top rows of Table 1 show the performance of RotoGrad and its both components in isolation.
All the methods with the same number of parameters.
Compared to Vanilla optimization (4th row), Rotate Only improves all metrics by homogenizing gradient directions.
Scale Only avoids overlooking the normal estimation task and improves on segmentation by homogenizing gradient magnitudes, at the expense of higher depth estimation error.
Remarkably, RotoGrad exploits its scaling and rotation components to obtain the best results in semantic segmentation and depth estimation, while still achieving comparable performance in the normal estimation task.
6. S UBSPACE ROTATIONS
We now evaluate the effect of subspace rotations as described at the end of §3.4, assessing the trade-off between avoiding negative transfer and size of the subspace considered by RotoGrad.
Experimental setup.
We test RotoGrad on a 10-task classiﬁcation problem on CIFAR10 (Krizhevsky et al., 2009), using binary cross-entropy and f1-score as loss and metric, respectively, for all tasks.
We use ResNet18 (He et al., 2016) without pre-training as backbone ( d = 512 ), and linear layers with sigmoid activation functions as task-speciﬁc heads.
[Table]
Results.
Table 2 (top) shows that rotating the entire space provides the best results
[Table]
and that these worsen as we decrease the size of R k .
Remarkably, rotating only 128 features already outperforms vanilla with no extra per-task parameters (1st row); and rotating 256 features already yields comparable results to vanilla optimization with extra capacity (6th row) despite its larger number of (task-speciﬁc) parameters.
These results can be further explained by Figure 9 in Appendix C.2, which shows a positive correlation between the size of R k and cosine similarity.
6. M ETHODS COMPARISON
Experimental setup.
In order to provide fair comparisons among methods, all experiments use identical conﬁgurations and random initializations.
For all methods we performed a hyperparameter search and chose the best ones based on validation error.
Unless otherwise speciﬁed, all baselines use the same architecture (and thus, number of parameters) as RotoGrad , taking each rotation matrix R k as extra task-speciﬁc parameters.
Further experimental details can be found in Appendix C.1, as well as extra experiments and complete results in Appendix C.2.
NYUv2.
Table 1 shows the performance of all baselines with and without the extra capacity.
RotoGrad signiﬁcantly improves performance on all tasks compared with vanilla optimization, and outperforms all other baselines.
Remarkably, we rotate only 1024 dimensions of z (out of a total of 7 millions) and, as a result, RotoGrad stays on par in training time with the baselines (around 4 h , Appendix C.2).
We can also assert the importance of learning the matrices
R k properly by comparing in Table 1 the different baselines with and without extra capacity.
This comparison reveals that the extra parameters do not solve the negative transfer but instead ampliﬁes biases (methods that overlook a subset of tasks, keep overlooking them) and, in the best case, provides trade-off solutions (also shown in
Appendix C.2).
Note, moreover, that RotoGrad (due to its Rotate Only component) is the only method to tackle conﬂicting gradient directions that manages to not overlook the normal surfaces task.
CIFAR10.
We reuse the setting from §6.
to compare different MTL baselines in terms of relative improvements (Table 2)
[Table] [Table]
and cosine similarity (Fig. 4), averaged over ﬁve different runs.
Table 2 (bottom) shows that, similar to the NYUv2 results, both direction-aware solutions (PCGrad and GradDrop) behave similar to vanilla optimization, marginally increasing the average improvement.
Unlike previous experiments, all magnitude-aware methods substantially worsen (at least) one of the statistics.
In contrast, RotoGrad improves the relative task improvement across all statistics using the same number of parameters.
Figure 4 shows the cosine similarity between task and update gradients, averaged over all tasks and runs (shaded areas correspond to 90 % conﬁdence intervals).
It is clear that RotoGrad obtains the highest cosine similarity, that other direction-aware methods also effectively align task gradients and, combined with the low cosine similarity achieved by MGDA-UB, suggests that there exists a correlation between cosine similarity and performance .
CelebA. To ﬁnalize, we test all methods in a 40-class multi-classiﬁcation problem on CelebA (Liu et al., 2015) and two different settings: one using a convolutional network as backbone ( d = 512 ); and another using ResNet18 (He et al., 2016) as backbone ( d = 2048 ).
As above, we use binary cross-entropy and f1-score as loss and metric for all tasks, thus accounting for highly imbalanced attributes.
Results in Table 3 show that RotoGrad performs great in all f1-score statistics and both architectures, specially in the convolutional neural network, outperforming competing methods.
Moreover, RotoGrad achieves these results rotating 50 % of the shared feature z for the convolutional network, and 75 % for the residual network, which further demonstrates that RotoGrad can scale-up to real-world settings.
We believe it is important to remark that, due to the high number of tasks, this setup is specially demanding.
Results in Appendix C.2 show the performance of all baselines without the rotation matrices, demonstrating the negative effect that the extra capacity can have if not learned properly, as well as that RotoGrad stays on par with non-extended baselines in training time.
C ONCLUSIONS
In this work, we have introduced RotoGrad, an algorithm that tackles negative transfer in MTL by homogenizing task gradients in terms of both magnitudes and directions.
RotoGrad enforces a similar convergence rate for all tasks, while at the same time smoothly rotates the shared representation differently for each task in order to avoid conﬂicting gradients.
As a result, RotoGrad leads to stable and accurate MTL.
Our empirical results have shown the effectiveness of RotoGrad in many scenarios, staying on top of all competing methods in performance, while being on par in terms of computational complexity with those that better scale to complex networks.
We believe our work opens up interesting venues for future work.
For example, it would be interesting to study alternative approaches to further scale up RotoGrad using, for example, diagonal-block or sparse rotation matrices; to rotate the feature space in application domains with structured features (e.g., channel-wise rotations in images); and to combine different methods, for example, by scaling gradients using the direction-awareness of IMTL-G and the “favor slow-learners” policy of RotoGrad.
Appendices A P ROOFS
Proposition A.1.
Suppose f k :=
L k ◦ h k is an inﬁnitely differentiable real-valued function, and let us call G k = ∇
Z f k ( Z ) its derivative with respect to Z , for every k = 1 , 2 , . . .
, K .
If cos _ sim( G i , G j ) > − 1 / ( K − 1) pairwise; then there exists a small-enough step size ε >such that, for all k , we have that L k ( h k ( Z − ε · C (cid:80)
k
U k ; φ k );
Y k )
<
L k ( h k ( Z ; φ k );
Y k ) , where
U k :=
G k / ||
G k || and C ≥ 0 .
Proof.
Since f k is inﬁnitely differentiable, we can take the ﬁrst-order Taylor expansion of f k around
Z , for any k , evaluated at Z − ε V for a given vector V :
[Equation]
In our case, V = C (cid:80) k
U k with C ≥ 0 , then:
[Equation]
Since || U k
||
= 1 for all k = 1 , 2 , . .
, K , it holds that − 1 ≤ cos _ sim( U k , U i ) =
(cid:104)
U k , U i (cid:105) ≤ 1 .
If cos _ sim( G k , G i ) > − 1 / ( K − 1) for all i (cid:54) =
k , then we have that 1 + (cid:80) i (cid:54) =
j (cid:104)
U k , U i (cid:105) > 0 and f k ( Z − ε V ) <
f k ( V ) for a small enough ε > 0 .
Q.E.D.
B S TACKELBERG GAMES AND R OTO G RAD
In game theory, a Stackelberg game (Fiez et al., 2020) is an asymmetric game where two players play alternately.
One of the players—whose objective is to blindly minimize their loss function—is known as the follower, F .
The other player is known as the leader, L .
In contrast to the follower, the leader attempts to minimize their own loss function, but it does so with the advantage of knowing which will be the best response to their move by the follower.
The problem can be written as
[Equation]
where x l ∈ X l and x f ∈ X f are the actions taken by the leader and follower, respectively.
While traditionally one assumes that players make perfect alternate moves in each step of problem 10, gradient-play Stackelberg games assume instead that players perform simultaneous gradient updates,
[Equation]
(11) where α l and α f are the learning rates of the leader and follower, respectively.
An important concept in game theory is that of an equilibrium point, that is, a point in which both players are satisﬁed with their situation, meaning that there is no available move immediately improving any of the players’ scores, so that none of the players is willing to perform additional actions/updates.
Speciﬁcally, we focus on the following deﬁnition introduced by Fiez et al.
Deﬁnition B.1 (differential Stackelberg equilibrium) .
A pair of points x ∗ l ∈ X l , x ∗ f ∈ X f , where x ∗ f = r
( x ∗ l ) is implicitly deﬁned by ∇ x f F ( x ∗ l , x ∗ f ) = 0 , is a differential Stackelberg equilibrium point if ∇ x l L ( x ∗ l , r ( x ∗ l ))
= 0 , and ∇ 2 x l L ( x ∗ l , r ( x ∗ l )) is positive deﬁnite.
Note that, when the players manage to reach such an equilibrium point, both of them are in a local optimum.
Here, we make use of the following result, introduced by Fiez et al.
(2020), to provide theoretical convergence guarantees to an equilibrium point:
Proposition B.1.
In the given setting, if the leader’s learning rate goes to zero at a faster rate than the follower’s, that is, α l ( t ) =
o ( α f ( t )) , where α i ( t ) denotes the learning rate of player i at step t , then they will asymptotically converge to a differential Stackelberg equilibrium point almost surely.
In other words, as long as the follower learns faster than the leader, they will end up in a situation where both are satisﬁed.
Even more, Fiez et al.
(2020) extended this result to the ﬁnite-time case, showing that the game will end close to an equilibrium point with high probability.
As we can observe, the Stackelberg formulation in Equation (10) is really similar to RotoGrad’s formulation in Equation (4).
Even more, the update rule in Equation (11) is quite similar to that one shown in Algorithm 1.
Therefore, it is sensible to cast RotoGrad as a Stackelberg game.
One important but subtle bit about this link regards the extra information used by the leader.
In our case, this extra knowledge explicitly appears in Equation 3 in the form of the follower’s gradient, g i,k , which is the direction the follower will follow and, as it is performing ﬁrst-order optimization by assumption, this gradient directly encodes the follower’s response.
Thanks to the Stackelberg formulation in Equation 4 we can make use of Prop.
B.1 and, thus, draw theoretical guarantees on the training stability and convergence.
In other words, we can say that performing training steps as those described in Algorithm 1 will stably converge as long as the leader is asymptotically the slow learner, that is α tl =
o ( α tf ) , where o denotes the little-o notation.
In practice, however, the optimization procedure proposed by Fiez et al.
(2020) requires computing the gradient of a gradient, thus incurring a signiﬁcant overhead.
Instead, we use Gradient Ascent-Descent (GDA), which only computes partial derivatives and enjoys similar guarantees (Jin et al., 2020), as we empirically showed in the manuscript.
C E XPERIMENTS C. E XPERIMENTAL SETUPS
Here, we discuss common settings across all experiments.
Refer to speciﬁc sections further below for details concerning single experiments.
Computational resources.
All experiments were performed on a shared cluster system with two
NVIDIA DGX-A100.
Therefore, all experiments were run with (up to) 4 cores of AMD EPYC
CPUs and, for those trained on GPU (CIFAR10, CelebA, and NYUv2), a single NVIDIA A100 GPU.
All experiments were restricted to 12 GB of RAM.
Loss normalization.
Similar as in the gradient case studied in this work, magnitude differences between losses can make the model overlook some tasks.
To overcome this issue, here we perform loss normalization, that is, we normalize all losses by their value at the ﬁrst training iteration (so that they are all 1 in the ﬁrst iteration).
To account for some losses that may quickly decrease at the start, after the 20th iteration, we instead normalize losses dividing by their value at that iteration.
Checkpoints.
For the single training of a model, we select the parameters of the model by taking those that obtained the best validation error after each training epoch.
That is, after each epoch we evaluate the linearly-scalarized validation loss, (cid:80)
k L k , and use the parameters that obtained the best error during training.
This can be seen as an extension of early-stopping where, instead of stopping, we keep training until reaching the maximum number of epochs hoping to keep improving.
Baselines.
We have implemented all baselines according to the original paper descriptions, except for PCGrad, which we apply to the gradients with respect to the feature z (instead of the shared parameters θ , as in the original paper).
Note that this is in accordance with recent works, for example
Chen et al.
(2020) and Liu et al.
(2021b), which also use this implementation of PCGrad in the feature space.
This way, all competing methods modify gradients with respect to the same variables and, as backpropagation performs the sum of gradients at the last shared representation z , PCGrad can be applied to reduce conﬂict at that level.
Hyperparameter tuning.
Model-speciﬁc hyperparameters were mostly selected by a combination of: i) using values described in prior works; and ii) empirical validation on the vanilla case (without any gradient-modiﬁers) to verify that the combinations of hyperparameters work.
To select methodspeciﬁc hyperparameters we performed a grid search, choosing those combinations of values that performed the best with respect to validation error.
Speciﬁcally, we took α ∈ { 0 , 0 . 5 , 1 , 2 } and R k ∈ R m × m with m ∈ { 0 .
25 d, 0 .
5 d, 0 .
75 d, d } (restricting ourselves to m ≤ 1500 ) for RotoGrad.
Regarding the learning rate of RotoGrad (GradNorm), we performed a grid search considering η roto ∈ { 0 .
05 η, 0 .
1 η, 0 .
5 η, η, 2 η } , where η roto and η are the learning rates of RotoGrad (GradNorm) and the network, respectively.
Statistical test.
For the tabular data, we highlight those results that are signiﬁcantly better than those from the multitask baseline (that is, better than the vanilla MTL optimization without the R k matrices).
To ﬁnd these values, we run a paired one-sided Student’s t-test across each method and the baseline.
For those metrics for which higher is better, our null hypothesis is that the method’s performance is equal or lower than the baseline, and for those for which lower is better, the null hypothesis is that the method’s performance is equal or greater than the baseline.
We use a signiﬁcance level of α = 0 . 05 .
Notation.
Along this section, we use the following to describe different architectures:
[Conv- F - C ] denotes a convolutional layer with ﬁlter size F and C number of channels; [Max] denotes a max-pool layer of ﬁlter size and stride 2, and [Dense- H ] a dense layer with output size H .
C.1. I LLUSTRATIVE EXAMPLES
Losses and metrics.
Both illustrative experiments use MSE as both loss and metric.
Regarding the speciﬁc form of ϕ in Equation (5), the avocado-shaped experiments uses
[Equation]
while the non-convex second experiment uses
[Equation]
Model.
As described in the main manuscript, we use a single input x ∈ R 2 picked at random from a standard normal distribution, and drop all task-speciﬁc network parameters (that is, h k ( r k )
=
r k ).
As backbone, we take a simple network of the form z = W 2 max( W 1 x + b 1 , 0) + b 2 with b 1 ∈ R 10 , b 2 ∈ R 2 , and W 1 , W (cid:62) 2 ∈ R 10 × 2 .
Hyperparameters, convex-case.
We train the model for one hundred epochs.
As network optimizer we use stochastic gradient descent (SGD) with a learning rate of 0 . 01 .
For the rotations we use
RAdam (Liu et al., 2019a) with a learning rate of 0 . 5 (for visualization purposes we need a high learning rate, in such a simple scenario it still converges) and exponential decay with decaying factor 0 .
Hyperparameter, non-convex case.
For the second experiment, we train the model for 400 epochs and, once again, use SGD as the network optimizer with a learning rate of 0 . 015 .
For the rotations, we use RAdam (Liu et al., 2019a) with a learning rate of 0 . 1 and an exponential decay of 0 .
C.1. MNIST/SVHN
Datasets.
We use two modiﬁed versions of MNIST (LeCun et al., 2010) and SVHN (Netzer et al., 2011) in which each image has two digits, one on each side of the image.
In the case of MNIST, both of them are merged such that they form an overlapped image of 28 × 28 , as shown in Figure 5a.
Since
SVHN contains backgrounds, we simply paste two images together without overlapping, obtaining images of size 32 × 64 , as shown in Figure 5b.
Moreover, we transform all SVHN samples to grayscale.
Tasks, losses, and metrics.
In order to further clarify the setup used, here we describe in detail each task.
Speciﬁcally, we have:
[Figure]
• Left digit classiﬁcation.
Loss: negative log-likelihood (NLL).
Metric: accuracy (ACC).
• Right digit classiﬁcation.
Loss: NLL.
Metric: ACC.
• Parity of the product of digits, that is, whether the product of both digits gives an odd number (binary-classiﬁcation).
Loss: binary cross entropy (BCE).
Metric: f1-score (F1).
Sum of both digits (regression).
Loss: MSE.
Metric: MSE.
• Active pixels in the image, that is, predict the number of pixels with values higher than 0 .
5 , where we use pixels lying in the unit interval (regression).
Loss: MSE.
Metric: MSE.
Model.
Our backbone is an adaption from the original LeNet (LeCun et al., 1998) model.
Speciﬁcally, we use:
• MNIST.
[Conv- 5 - 10 ]
[Max][ReLU][Conv- 5 - 20 ][Max][Dense- 50 ][ReLU][BN] , • SVHN.
[Conv- 5 - 10 ]
[Max][ReLU][Conv- 5 - 20 ][Max][Conv- 5 - 20 ][Dense- 50 ] [ReLU][BN] ,
where [BN] refers to Batch Normalization (Ioffe & Szegedy, 2015).
Depending on the type of task, we use a different head.
Speciﬁcally, we use:
• Regression.
[Dense- 50 ]
[ReLU][Dense- 1 ] , • Classiﬁcation.
[Dense- 50 ]
[ReLU][Dense- 10 ][Log-Softmax] , • Binary-classiﬁcation.
[Dense- 1 ][Sigmoid] .
Model hyperparameters.
For both datasets, we train the model for 300 epochs using a batch size of 1024 .
For the network parameters, we use RAdam (Liu et al., 2019a) with a learning rate of 1 e − 3 .
Methods hyperparameters.
In Tables 4 and 5 we show the results of GradNorm with: i) MNIST with R k , α = 0 ; ii) MNIST without R k , α = 0 .
5 ; iii) SVHN with R k , α = 1 ; and iv) SVHN without
R k , α = 2 .
We train RotoGrad with full-size rotation matrices ( m = d ).
Both methods use RAdam with learning rate 5 e − 4 and exponential decay of 0 . 9999 .
C.1. CIFAR
Dataset.
We use CIFAR10 (Krizhevsky et al., 2009) as dataset, with 40 000 instances as training data and the rest as testing data.
Additionally, every time we get a sample from the dataset we: i) crop the image by a randomly selected square of size 3 × 32 × 32 ; ii) randomly ﬂip the image horizontally; and iii) standardize the image channel-wise using the mean and standard deviation estimators obtained on the training data.
Model.
We take as backbone ResNet18 (He et al., 2016) without pre-training, where we remove the last linear and pool layers.
In addition, we add a Batch Normalization layer.
For each task-speciﬁc head, we simply use a linear layer followed by a sigmoid function, that is, [Dense- 1 ][Sigmoid] .
Losses and metrics.
We treat each class (out of ten) as a binary-classiﬁcation task where we use
BCE and F1 as loss and metric, respectively.
Model hyperparameters.
We use a batch size of 128 and train the model for 500 epochs.
For the network parameters, we use as optimizer SGD with learning rate of 0 .
01 , Nesterov momentum of 0 . 9 , and a weight decay of 5 e − 4 .
Additionally, we use for the network parameters a cosine learning-rate scheduler with a period of 200 iterations.
Methods hyperparameters.
Results shown in Tables 2 and 6 use α = 0 and α = 0 .
5 for GradNorm with and without R k , respectively, and we use RAdam (Liu et al., 2019a) as optimizer with learning rate 0 .
001 and an exponential decay factor of 0 .
999 95 for both GradNorm and RotoGrad.
C. . NYU V
Setup.
In contrast with the rest of experiments, for the NYUv2 experiments shown in §6, instead of writing our own implementation, we slightly modiﬁed the open-source code provided by Liu et al.
(2019b) at https://github.com/lorenmt/mtan (commit 268c5c1).
We therefore use the exact same setting as Liu et al.
(2019b)—and refer to their paper and code for further details, with the addition of using data augmentation for the experiments which, although not described in the paper, is included in the repository as a command-line argument.
We will provide along this work a diff ﬁle to include all gradient-modiﬁer methods into the aforementioned code.
Methods hyperparameters.
For the results shown in Table 1 we use GradNorm with α = 0 and
RotoGrad with rotations R k of size 1024 .
We use a similar optimization strategy as the rest of parameters, using Adam (Kingma & Ba, 2014) with learning rate 5 e − 5 (half the one of the network parameters) and where we halve this learning rate every 100 iterations.
C.1. C ELEB A
Dataset.
We use CelebA (Liu et al., 2015) as dataset with usual splits.
We resize each sample image so that they have size 3 × 64 × 64 .
Losses and metrics.
We treat each class (out of forty) as a binary-classiﬁcation task where we use
BCE and F1 as loss and metric, respectively.
ResNet model.
As with CIFAR10, we use as backbone ResNet18 (He et al., 2016) without pretraining, where we remove the last linear and pool layers.
In addition, we add a Batch Normalization layer.
For each task-speciﬁc head, we use a linear layer followed by a sigmoid function, that is, [Dense- 1 ][Sigmoid] .
ResNet hyperparameters.
We use a batch size of 256 and train the model for 100 epochs.
For the network parameters, we use RAdam (Liu et al., 2019a) as optimizer with learning rate 0 .
001 and exponential decay of 0 .
999 95 applied every 2400 iterations.
Convolutional model.
For the second architecture, we use a convolutional network as backbone, [Conv- 3 - 64 ]
[BN][Max][Conv- 3 - 128 ][BN][Conv- 3 - 128 ][BN][Max][Conv- 3 - 256 ][BN] [Conv- 3 - 256 ][BN][Max][Conv- 3 - 512 ][BN][Dense- 512 ][BN] .
For the task-speciﬁc heads, we take a simple network of the form [Dense- 128 ]
[BN][Dense- 1 ][Sigmoid] .
Convolutional hyperparameters.
We use a batch size of 8 and train the model for 20 epochs.
For the network parameters, we use RAdam (Liu et al., 2019a) as optimizer with learning rate 0 .
001 and exponential decay of 0 .
96 applied every 2400 iterations.
Methods hyperparameters.
Results shown in Tables 3 and 7 use GradNorm with: i) convolutional network with R k , α = 0 ; ii) convolutional network without R k , α = 1 ; iii) residual network with
R k , α = 0 . 5 ; and iv) residual network without R k , α = 1 .
For RotoGrad, we rotate 256 andelements of z for the convolutional and residual networks.
As optimizer, we use RAdam (Liu et al., 2019a) with learning rate 5 e − 6 and an exponential decay factor of 0 .
999 95 for both GradNorm and
RotoGrad.
Note that for these experiments we omit MGDA-UB (Sener & Koltun, 2018) as it is computationally prohibitive in comparisons with other methods.
In single-seed experiments, we however observed that it does not perform too well (specially in the convolutional network).
C. A DDITIONAL RESULTS C.2.
[Table]
ULTI
[Table]
[Table] [Table] We reuse the experimental setting from §6.1—now using the original LeNet (LeCun et al., 1998) and a multitask-version of SVHN (Netzer et al., 2011)—in order to evaluate how disruptive the orthogonal image-related task is for different methods.
We can observe (Table 4) that the effect of the image-related task is more disruptive in MNIST, in which MGDA-UB utterly fails.
Direction-aware methods (GradDrop and PCGrad) do not improve the vanilla results, whereas IMTL-G, GradNorm, and RotoGrad obtain the best results.
We also provide the complete results for all metrics in Table 5.
In the case of MNIST, we can observe that both regression tasks tend to be quite disruptive.
GradNorm, IMTL-G, and RotoGrad manage to improve over all tasks while maintaining good performance on the rest of tasks.
MGDA-UB, however, focuses on the image-related task too much and overlooks other tasks.
In SVHN we observe a similar behavior.
This time, all methods are able to leverage positive transfer and improve their results on the parity and sum tasks, obtaining similar task improvement results.
Yet, the image-related task is more disruptive than before, showing bigger differences between methods.
As before, MGDA-UB completely focuses on this task, but now is able to not overlook any task while doing so.
Regarding the rest of the methods, all of them improved their results with respect to the vanilla case, with
RotoGrad and GradNorm obtaining the second-best results.
[Table]
[Figure]
We complement the illustrative ﬁgures shown in Figure 1 by providing, for each example, an illustration of the effect of
RotoGrad shown as an active and passive transformation.
In an active transformation (Figure 6 left), points in the space are the ones modiﬁed.
In our case, this means that we rotate feature z , obtaining r 1 and r 2 , while the loss functions remain the same.
In other words, for each z we obtain a taskspeciﬁc feature r k that optimizes its loss function.
In contrast, a passive transformation (Figure 6 right) keeps the points unaltered while applying the transformation to the space itself.
In our case, this translates to rotating the optimization landscape of each loss function (now we have L k ◦ R k instead of L K ), so that our single feature z has an easier job at optimizing both tasks.
In the case of RotoGrad, we can observe in both right ﬁgures that both optima lie in the same point, as we are aligning task gradients.
[Figure]
RotoGrad learns both opposite rotations R 1 =
R (cid:62) 2 .
Besides the two regression experiments shown in §4, we include here an additional experiment where we test RotoGrad in the worst-case scenario of gradient conﬂict, that is, one in which task gradients are opposite to each other.
To this end, we solve a 2-task binary classiﬁcation problem where, as dataset, we take 1000 samples from a 2D Gaussian mixture model with two clusters; y n,k =
if x n was sampled from cluster k ; and y n,k = 0 otherwise.
We use as model a logistic regression model of the form y k
= W 2 max( W 1 x + b 1 , 0)
+ bwith b 1 ∈ R 2 , b 2 ∈ R , W 1 ∈ R 2 × 2 , and
W 2 ∈ R 1 × 2 .
Because rotations in 1D are ill-posed (there is a unique proper rotation), here we add task parameters to increase the dimensionality of z and make all parameters shared , so that there is still no task-speciﬁc parameters.
To avoid a complete conﬂict where ∇ z L 1 + ∇ z L 2 = 0 , we randomly ﬂip the labels for the second tasks with 5 % probability.
Figure 7 shows that, in this extreme scenario,
RotoGrad is able to learn both tasks by aligning gradients, that is, by learning that one rotation is the inverse of the other R 1 =
R (cid:62) 2 .
C.2. T RAINING STABILITY
While we showed in §6.1 only the results for the sum-of-digits task as they were nice and clear, here we show in Figure 8 the results of those same experiments in §6.1 for all the different tasks.
The same discussion from the main manuscript can be carried out for all metrics.
Additionally, we can observe that the vanilla case (learning rate zero) completely overlooks the image-related task ( Active pixels ) while performing the best in the parity task.
Additionally, let us clarify what we mean here with stability, as in the main manuscript we mainly talked about convergence guarantees.
In these experiments we measure the convergence guarantees of the experiments in terms of ‘training stability’, meaning the variance of the obtained results across different runs.
The intuition here is that, since the model does not converge, we should expect some wriggling learning curves during training and, as we take the model with the best validation error, the individual task metrics should have bigger variance (that is, less stability) across runs
[Figure]
all tasks as we highly increase/decrease RotoGrad’s learning rate, as discussed in the main manuscript.
C.2. CIFAR10 AND C ELEB A
For the sake of completeness, we present in Table 6 and Table 7 the same tables as in §6, but with more statistics of the results.
For CIFAR10, we now included in Table 6 the minimum task improvement across tasks and, while noisier, we can still observe that RotoGrad also improve this statistic.
The standard deviation of the task improvement across tasks is, however, not too informative.
We also include in Figure 9 the cosine similarity plots for the different subspace rotations from §6.3, showing a clear positive correlation between the cosine similarity and the size of the considered subspace.
While the cosine similarities look low, we want to remark that we are computing the cosine similarity of a huge space, and we only align a subspace of it.
If we, instead, showed the cosine similarity with respect to each speciﬁc subspace, the cosine similarity should look similar to that of RotoGrad 512.
In the case of CelebA, we added in Table 7 the maximum f1-score across tasks and, similar to the last case, it is not too informative, as all methods achieve almost perfect f1-score in one of the classes.
We also include the training times for some baselines, showing that RotoGrad stays on par with them.
C.2. NYU V
Complementing the results shown in §6, we show in Table
the results obtained combining RotoGrad with all other existing methods (rows within RotoGrad + ), for gradient scaling methods we only apply the rotation part of RotoGrad.
Results show that RotoGrad helps improve/balance all other
[Table]
[Table]
[Table]
RotoGrad
[Table]
Vanilla
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
W
[Table]
[Table]
[Table]
GradDrop
[Table]
methods, which is specially true for those methods that heavily overlook some tasks.
Speciﬁcally,
MGDA-UB stops overlooking the semantic segmentation and depth estimation tasks, while PCGrad and GradDrop stop completely overlooking the surface normal loss.
Note that we also show in
Table 8 the training times of each method, and RotoGrad stays on par with non-extended methods in training time.
As mentioned in Appendix C.1, due to cluster overload, some times were deceivingly high (speciﬁcally those baselines with R k ) as we had to run them on different machines, and were omitted to avoid confusion.