U NDERSTANDING AND I MPROVING I NFORMATION T RANSFER IN M ULTI -T ASK L EARNING Anonymous authors
Paper under double-blind review
A BSTRACT
We investigate multi-task learning approaches which use a shared feature representation for all tasks.
To better understand the transfer of task information, we study an architecture with a shared module for all tasks and a separate output module for each task.
We study the theory of this setting on linear and ReLU-activated models.
Our key observation is that whether or not tasks’ data are well-aligned can signiﬁcantly affect the performance of multi-task learning.
We show that misalignment between task data can cause negative transfer (or hurt performance) and provide sufﬁcient conditions for positive transfer.
Inspired by the theoretical insights, we show that aligning tasks’ embedding layers leads to performance gains for multi-task training and transfer learning on the GLUE benchmark and sentiment analysis tasks; for example, we obtain a 2.35% GLUE score average improvement on 5 GLUE tasks over BERT LARGE using our alignment method.
We also design an SVD-based task reweighting scheme and show that it improves the robustness of multi-task training on a multi-label image dataset.
I NTRODUCTION
Multi-task learning has recently emerged as a powerful paradigm in deep learning to obtain language (Devlin et al.
; Liu et al.
(2019a;b))
and visual representations (Kokkinos (2017)) from large-scale data.
By leveraging supervised data from related tasks, multi-task learning approaches reduce the expensive cost of curating the massive per-task training data sets needed by deep learning methods and provide a shared representation which is also more efﬁcient for learning over multiple tasks.
While in some cases, great improvements have been reported compared to single-task learning (McCann et al.
, practitioners have also observed problematic outcomes, where the performances of certain tasks have decreased due to task interference (Alonso and Plank (2016); Bingel and Søgaard (2017)).
Predicting when and for which tasks this occurs is a challenge exacerbated by the lack of analytic tools.
In this work, we investigate key components to determine whether tasks interfere constructively or destructively from theoretical and empirical perspectives.
Based on these insights, we develop methods to improve the effectiveness and robustness of multi-task training.
There has been a large body of algorithmic and theoretical studies for kernel-based multi-task learning, but less is known for neural networks.
The conceptual message from the earlier work (Baxter (2000); Evgeniou and Pontil (2004); Micchelli and Pontil (2005); Xue et al.
(2007)) show that multi-task learning is effective over “similar” tasks, where the notion of similarity is based on the single-task models (e.g. decision boundaries are close).
The work on structural correspondence learning (Ando and Zhang (2005); Blitzer et al.
(2006)) uses alternating minimization to learn a shared parameter and separate task parameters.
Zhang and Yeung (2014) use a parameter vector for each task and learn task relationships via l 2 regularization, which implicitly controls the capacity of the model.
These results are difﬁcult to apply to neural networks: it is unclear how to reason about neural networks whose feature space is given by layer-wise embeddings.
To determine whether two tasks interfere constructively or destructively, we investigate an architecture with a shared module for all tasks and a separate output module for each task (Ruder (2017)).
See Figure 1 for an illustration.
Our motivating observation is that in addition to model similarity which affects the type of interference, task data similarity plays a second-order effect after controlling model similarity.
To illustrate the idea, we consider three tasks with the same number of data samples where task 2 and 3 have the same decision boundary but different data distributions (see
Figure 2 for an illustration).
We observe that training task 1 with task 2 or task 3 can either improve or hurt task 1’s performance, depending on the amount of contributing data along the decision
[Figure] [Figure]
boundary!
This observation shows that by measuring the similarities of the task data and the models separately, we can analyze the interference of tasks and attribute the cause more precisely.
Motivated by the above observation, we study the theory of multi-task learning through the shared module in linear and ReLU-activated settings.
Our theoretical contribution involves three components: the capacity of the shared module , task covariance , and the per-task weight of the training procedure .
The capacity plays a fundamental role because, if the shared module’s capacity is too large, there is no interference between tasks; if it is too small, there can be destructive interference.
Then, we show how to determine interference by proposing a more ﬁne-grained notion called task covariance which can be used to measure the alignment of task data.
By varying task covariances, we observe both positive and negative transfers from one task to another!
We then provide sufﬁcient conditions which guarantee that one task can transfer positively to another task, provided with sufﬁciently many data points from the contributor task.
Finally, we study how to assign per-task weights for settings where different tasks share the same data but have different labels.
Our theory leads to the design of two algorithms with practical interest.
First, we propose to align the covariances of the task embedding layers and present empirical evaluations on well-known benchmarks and tasks.
On 5 tasks from the General Language Understanding Evaluation (GLUE) benchmark (Wang et al. (2018b)) trained with the BERT LARGE model by Devlin et al.
(2018), our method improves the result of BERT LARGE by a 2.35% average GLUE score, which is the standard metric for the benchmark.
Further, we show that our method is applicable to transfer learning settings; we observe up to 2.5% higher accuracy by transferring between six sentiment analysis tasks using the
LSTM model of Lei et al.
Second, we propose an SVD-based task reweighting scheme to improve multi-task training for settings where different tasks have the same data but different labels.
On the ChestX-ray14 image classiﬁcation dataset, we compare our method to the unweighted scheme and observe an improvement of 5.6 AUC score for all tasks.
In conclusion, these evaluations conﬁrm that our theoretical insights are applicable to a broad range of settings and applications.
T HREE C OMPONENTS OF M ULTI -T ASK L EARNING
We study multi-task learning (MTL) models with a shared module for all tasks and a separate output module for each task.
We ask: What are the key components to determine whether or not MTL is better than single-task learning (STL)?
In response, we identify three components: model capacity , task covariance , and optimization scheme .
After setting up the model, we brieﬂy describe the role of model capacity.
We then introduce the notion of task covariance , which comprises the bulk of the section.
We ﬁnish by showing the implications of our results for choosing optimization schemes.
2. M ODELING S ETUP
We are given k tasks.
Let m i denote the number of data samples of task i .
For task i , let X i ∈
R m i ×
d denote its covariates and let y i ∈
R m i denote its labels, where d is the dimension of the data.
We have assumed that all the tasks have the same input dimension d .
This is not a restrictive assumption and is typically satisﬁed, e.g. for word embeddings on BERT, or by padding zeros to the input otherwise.
Our model assumes the output label is 1-dimensional.
We can also model a multi-label problem with k types of labels by having k tasks with the same covariates but different labels.
We consider an MTL model with a shared module B ∈ R d × r and a separate output module A i ∈ R r for task i , where r denotes the output dimension of B .
See Figure 1 for the illustration.
We deﬁne the objective of ﬁnding an MTL model as minimizing the following equation over B and the A i ’s:
[Equation]
where L is a loss function such as the squared loss.
The activation function g : R → R is applied on every entry of X i B .
In equation 1, all data samples contribute equally.
Because of the differences between tasks such as data size, it is natural to re-weight tasks during training:
[Equation]
This setup is an abstraction of the hard parameter sharing architecture (Ruder (2017)).
The shared module B provides a universal representation (e.g., an LSTM for encoding sentences) for all tasks.
Each task-speciﬁc module A i is optimized for its output.
We focus on two models as follows.
The single-task linear model.
The labels y of each task follow a linear model with parameter θ ∈ R
d
y = Xθ + ε .
Every entry of ε follows the normal distribution N (0 , σ 2 ) with variance σ 2 .
The function g ( XB ) = XB .
This is a well-studied setting for linear regression (Hastie et al.
The single-task ReLU model.
Denote by ReLU ( x ) = max( x, 0) for any x ∈ R .
We will also consider a non-linear model where Xθ goes through the ReLU activation function with a ∈ R and θ ∈ R d : y = a · ReLU ( Xθ )+ ε , which applies the ReLU activation on Xθ entrywise.
The encoding function g ( XB ) then maps to ReLU ( XB ) .
Positive vs. negative transfer.
For a source task and a target task, we say the source task transfers positively to the target task, if training both through equation 1 improves over just training the target task (measured on its validation set).
Negative transfer is the converse of positive transfer.
Problem statement.
Our goal is to analyze the three components to determine positive vs. negative transfer between tasks: model capacity ( r ), task covariances ( { X (cid:62) i X i } ki =1 ) and the per-task weights ( { α i } ki =1 ).
We focus on regression tasks under the squared loss but we also provide synthetic experiments on classiﬁcation tasks to validate our theory.
Notations.
For a matrix X , its column span is the set of all linear combinations of the column vectors of X .
Let X † denote its psuedoinverse.
Given u, v ∈ R d , cos( u, v ) is equal to u (cid:62)
v/
( (cid:107) u (cid:107) · (cid:107) v (cid:107) ) .
2. M ODEL C APACITY
We begin by revisiting the role of model capacity, i.e. the output dimension of B (denoted by r ).
We show that as a rule of thumb, r should be smaller than the sum of capacities of the STL modules.
Example.
Suppose we have k linear regression tasks using the squared loss, equation 1 becomes:
[Equation]
The optimal solution of equation 3 for task i is θ i =
( X (cid:62) i X i ) † X (cid:62) i
y i
∈
R d .
Hence a capacity ofsufﬁces for each task.
We show that if r ≥ k , then there is no transfer between any two tasks.
Proposition 1.
Let r ≥ k .
There exists an optimum B (cid:63) and { A (cid:63)i } ki =1 of equation 3 where B (cid:63) A (cid:63)i = θ i , for all
i = 1 , 2 , . . .
, k .
To illustrate the idea, as long as B (cid:63) contains { θ i } ki =1 in its column span, there exists A (cid:63)i such that
B (cid:63) A (cid:63)i = θ i , which is optimal for equation 3 with minimum error.
But this means no transfer among any two tasks.
This can hurt generalization if a task has limited data, in which case its STL solution overﬁts to the training data, whereas the MTL solution can leverage other tasks’ data to improve generalization.
The proof of Proposition 1 and its extension to ReLU settings are in Appendix B.1.
Algorithmic consequence.
The implication is that limiting the shared module’s capacity is necessary to enforce information transfer.
In practice, if the shared module is too small, then it interferes with task transfer.
But if it is too large, then no transfer occurs.
The ideal capacity depends on task data similarity (e.g. smaller for similar tasks), which leads to the question of how to quantify them.
[Figure]
2. T ASK C OVARIANCE
To show how to quantify task data similarity we illustrate with two regression tasks under the linear model without noise: y
=
X 1 θ 1 and y
=
X 2 θ 2 .
By Section 2.
it is necessary to limit the capacity of the shared module to enforce information transfer.
Therefore, we consider the case of r = 1 .
Hence, the shared module B is now a d -dimensional vector and A 1 , A 2 are both scalars.
A natural requirement of task similarity is for the STL models to be similar, i e. | cos( θ 1 , θ 2 ) | to be large.
To see this, the optimal STL model for task 1 is ( X (cid:62) 1 X 1 ) − 1 X (cid:62) 1 y
= θ 1 .
Hence if | cos( θ 1 , θ 2 ) | is 1, then tasks 1 and 2 can share a model B ∈ R d which is either θ 1 or − θ 1 .
The scalar A 1 and Acan then transform B to be equal to θ 1 and θ 2 .
Is this requirement sufﬁcient?
Recall that in equation 3, the task data X 1 and X 2 are both multiplied by B .
If they are poorly “aligned” geometrically, the performance could suffer.
How do we formalize the geometry between task alignment?
In the following, we show that the covariance matrices of
X 1 and X 2 , which we deﬁne to be X (cid:62) 1 X 1 and X (cid:62) 2 X 2 , captures the geometry.
We ﬁx | cos( θ 1 , θ 2 ) | to be close to 1 to examine the effects of task covariances.
In Appendix B.2.1 we ﬁx task covariances to examine the effects of model cosine similarity.
Concretely, equation 3 reduces to:
[Equation]
where we apply the ﬁrst-order optimality condition on A 1 and A 2 and simplify the equation.
Specifically, we focus on a scenario where task 1 is the source and task 2 is the target .
Our goal is to determine when the source transfers to the target positively or negatively in MTL.
Determining the type of transfer from task 2 to task 1 can be done similarly.
Answering the question boils down to studying the angle or cosine similarity between the optimum of equation 4 and θ 2 .
Example.
In Figure 3, we show that by varying task covariances and the number of samples, we can observe both positive and negative transfers.
The conceptual message is the same as Figure 2; we describe the data generation process in more detail.
We use 3 tasks and measure the type of transfer from the source to the target.
The x -axis is the number of data samples from the source.
The y -axis is the target’s performance improvement measured on its validation set between MTL minus STL.
Data generation.
We have | cos( θ 1 , θ 2 ) | ≈ 1 (say 0.96).
For i ∈ { 1 , 2 , 3 } , let R i ⊆
R m i ×
d denote a random Gaussian matrix drawn from N (0 , 1) .
Let S 1 , S 2 ⊆ { 1 , 2 , . . .
, d } be two disjoint sets of size d/ 10 .
For i = 1 , 2 , let D i be a diagonal matrix whose entries are equal to a large value κ (e.g. κ = 100 ) for coordinates in S i and 1 otherwise.
Let Q i ⊆
R d × d denote an orthonormal matrix, i.e.
Q (cid:62) i
Q i is equal to the identity matrix, orthogonalized from a random Gaussian matrix.
Then, we deﬁne the 3 tasks as follows.
(i) Task 1 (target): X 1 = R 1 Q 1 D 1 and y
=
X
θ 1 .
(ii)
Task 2 (source task for red line):
X 2 =
R 2 Q 1 D 1 and y 2 =
X
θ 2 .
(iii) Task 3 (source task for green line):
X 3 = R 3 Q 2 D 2 and y
= X 3 θ 2 .
Task 1 and 2 have the same covariance matrices but task 1 andhave different covariance matrices.
Intuitively, the signals of task 1 and 3 lie in different subspaces, which arise from the difference in the diagonals of D i and the orthonormal matrices.
Analysis.
Unless the source task has lots of samples to estimate θ 2 , which is much more than the samples needed to estimate only the coordinates of S 1 , the effect of transferring to the target is small.
We observe similar results for logistic regression tasks and for ReLU-activated regression tasks.
Require:
Task embedding layers X 1 ∈ R m 1 × d , X 2 ∈ R m 2 × d , . .
, X k ∈ R m k × d , shared module B
Parameter: Alignment matrices R 1 , R 2 , . . .
, R k ∈ R d × d and output modules
A 1 , A 2 . . .
, A k ∈ R r 1: Let Z i
=
X i R i , for 1 ≤ i ≤ k .
Consider the following modiﬁed loss (with B being ﬁxed):
ˆ f
( A 1 , . . .
, A k ; R 1 , . . .
, R k ) = (cid:80)
ki =1 L ( g ( Z i B )
A i , y i ) = (cid:80) ki
=1 L ( g ( X i R i B )
A i , y i ) 2: Minimize ˆ f jointly over R 1 , R 2 , . . .
, R k and the output layers A 1 , A 2 , . . .
, A k
Theory.
We rigorously quantify how many data points is needed to guarantee positive transfer.
The folklore in MTL is that when a source task has a lot of data but the related target task has limited data, then the source can often transfer positively to the target task.
Our previous example shows that by varying the source’s number of samples and its covariance, we can observe both types of transfer.
How much data do we need from the source to guarantee a positive transfer to the target?
We show that this depends on the condition numbers of both tasks’ covariances.
Theorem 2 (informal) .
For i = 1 , 2 , let y i
=
X i
θ i
+ ε i denote two linear regression tasks with parameters θ i ∈
R
d and m i number of samples.
Suppose that each row of the source task X 1 is drawn independently from a distribution with covariance Σ
1 ⊆ R d × d and bounded l 2 -norm.
Let c
= κ ( X 2 )sin( θ 1 , θ 2 ) and assume that c ≤ 1 / 3 .
Denote by ( B (cid:63) , A (cid:63) 1 , A (cid:63) 2 ) the optimal MTL solution.
With high probability, when m 1 is at least on the order of ( κ 2 (Σ 1 ) · κ 4 ( X 2 ) · (cid:107) y 2 (cid:107) 2 ) /c 4 , we have
[Equation]
Recall that for a matrix X , κ ( X ) denotes its condition number.
Theorem 2 quantiﬁes the trend in
Figure 3, where the improvements for task 2 reaches the plateau when m 1 becomes large enough.
As an example, if sin( θ 1 , θ 2 ) ≤ δ/κ ( X 2 ) , then equation 5 is at most O ( δ ) + (cid:107) ε 2 (cid:107) / (cid:107) X
2 θ 2 (cid:107) .
The formal statement, its proof and discussions on the assumptions are deferred to Appendix B.2.2.
The ReLU model.
We show a similar result for the ReLU model, which requires resolving the challenge of analyzing the ReLU function.
We use a geometric characterization for the ReLU function under distributional input assumptions by Du et al.
The result is deferred to Appendix B.2.3.
Algorithmic consequence.
An implication of our theory is a covariance alignment method to improve multi-task training.
For the i -th task, we add an alignment matrix R i before its input
X
i passes through the shared module B .
Algorithm 1 shows the procedure.
We also propose a metric called covariance similarity score to measure the similarity between two tasks.
Given X 1 ∈ R m 1 × d and X 2 ∈ R m 2 × d , we measure their similarity in three steps: (a) The covariance matrix is X (cid:62) 1 X 1 .
(b) Find the best rank- r 1 approximation to be U 1 ,r 1 D 1 ,r 1 U (cid:62) 1 ,r 1 , where r 1 is chosen to contain 99% c
Apply b compute
Covariance similarity score
[Equation]
The nice property of the score is that it is invariant to rotations of the columns of X 1 and X 2 .
2. O PTIMIZATION S CHEME
Lastly, we consider the effect of re-weighting the tasks (or their losses in equation 2).
When does reweighting the tasks help?
In this part, we show a use case for improving the robustness of multi-task training in the presence of label noise.
The settings involving label noise can arise when some tasks only have weakly-supervised labels, which have been studied before in the literature (e.g. Mintz et al.
; Pentina and Lampert (2017)).
We start by describing a motivating example.
Consider two tasks where task 1 is y 1 =
Xθ and task 2 is y
=
Xθ + ε 2 .
When we train the two tasks together, the error ε 2 will add noise to the trained model.
However, by up weighting task 1, we reduce the noise from task 2 and get better performance.
To rigorously study the effect of task weights, we consider a setting where all the tasks have the same data but different labels.
This setting arises for example in multi-label image datasets.
We study the linear model to show how the re-weighted scheme can change the optimal solution.
Proposition 3.
Let the shared module have capacity r ≤ k .
Given k tasks with the same covariates
X ⊆ R m × d but different labels { y i } ki =1 .
Let X be full rank and UDV (cid:62) be its SVD.
Let Q r Q (cid:62) r be the best rank- r approximation to (cid:80)
ki
=
α i U (cid:62)
y
i y (cid:62) i U .
Let B (cid:63) ⊆
R d × r be an optimal solution for the re-weighted loss.
Then the column span of B (cid:63) is equal to the column span of ( X (cid:62) X ) − 1 V DQ r .
We can also extend Proposition 3 to show that all local minima of equation 3 are global minima in the linear setting.
We leave the proof to Appendix B.3.
We remark that this result does not extend the non-linear ReLU setting.
Based on Proposition 3, we provide a rigorous proof of the previous example.
Suppose that X is full rank, ( X (cid:62) X ) † X
[ α 1 y 1 , α 1 y 2 ]) =
[ α 1 θ, α 2 θ + α 2 ( X (cid:62) X ) − 1 Xε 2 ] .
Hence, when we increase α 1 , cos( B (cid:63) , θ ) increases closer to 1.
[Table]
, for i = 1 , 2 , . . . , k .
Algorithmic consequence.
Inspired by our theory, we describe a re-weighting scheme in the presence of label noise.
We compute the per-task weights by computing the SVD over X (cid:62)
y i , for 1 ≤ i ≤ k .
The intuition is that if the label vector of a task y i is noisy, then the entropy of y i is small.
Therefore, we would like to design a procedure that removes the noise.
The SVD procedure does this, where the weight of a task is calculated by its projection into the principal r directions.
See Algorithm 2 for the description.
E XPERIMENTS
We describe connections between our theoretical results and practical problems of interest.
We show three claims on real world datasets.
(i)
The shared MTL module is best performing when its capacity is smaller than the total capacities of the single-task models.
(ii)
Our proposed covariance alignment method improves multi-task training on a variety of settings including the GLUE benchmarks and six sentiment analysis tasks.
Our method can be naturally extended to transfer learning settings and we validate this as well.
(iii) Our SVD-based reweighed scheme is more robust than the standard unweighted scheme on multi-label image classiﬁcation tasks in the presence of label noise.
3. E XPERIMENTAL S ETUP
Datasets and models.
We describe the datasets and models we use in the experiments.
GLUE: GLUE is a natural language understanding dataset including question answering, sentiment analysis, text similarity and textual entailment problems.
We choose BERT LARGE as our model, which is a 24 layer transformer network from Devlin et al.
Sentiment Analysis: This dataset includes six tasks: movie review sentiment (MR), sentence subjectivity (SUBJ), customer reviews polarity (CR), question type (TREC), opinion polarity (MPQA), and the Stanford sentiment treebank (SST) tasks.
For each task, the goal is to categorize sentiment opinions expressed in the text.
We use an embedding layer followed by an LSTM layer proposed by Lei et al.
We use the GloVe embeddings (http://nlp.stanford.edu/data/wordvecs/glove.6B.zip).
ChestX-ray14:
This dataset contains 112,120 frontal-view X-ray images and each image has up to 14 diseases.
This is a 14-task multi-label image classiﬁcation problem.
We use the CheXNet model from Rajpurkar et al.
(2017), which is a 121-layer convolutional neural network on all tasks.
For all models, we share the main module across all tasks (BERT LARGE for GLUE, LSTM for sentiment analysis, CheXNet for ChestX-ray14) and assign a separate regression or classiﬁcation layer on top of the shared module for each tasks.
Comparison methods.
For the experiment on multi-task training, we compare Algorithm 1 by training with our method and training without it.
Speciﬁcally, we apply the alignment procedure on the task embedding layers.
See Figure 4 for an illustration, where E i denotes the embedding of task
i , R i denotes its alignment module and Z i =
E i R i is the rotated embedding.
For transfer learning, we ﬁrst train an STL model on the source task by tuning its model capacity (e.g. the output dimension of the LSTM layer).
Then, we ﬁne-tune the STL model on the target task for 510 epochs.
To apply Algorithm 1, we add an alignment module for the target task during ﬁne-tuning.
𝐴 # 𝐴 $ 𝐴 % … 𝐴 &’$ 𝐴 &’# 𝐴 &
Task Embeddings
Shared Modules (LSTM/BERT) 𝑅 & 𝐼 # 𝑅 $ 𝑅 % … 𝑅 &’$ 𝑅 &’# 𝑋 # 𝑋 $ 𝑋 % … 𝑋 &’$ 𝑋 &’# 𝑋 & 𝐸 , 𝑍 ,
For the experiment on reweighted schemes, we ﬁrst compute the per-task weights as described in Algorithm 2.
Then, we reweight the loss function as in equation 2.
We compare with the reweighting techniques of Kendall et al.
Informally, the latter uses the Gaussian likelihood to model classiﬁcation outputs.
The weights, deﬁned as inversely proportional to the variances of the Gaussian, are optimized during training.
We also compare with the unweighted loss (cf.
equation 1) as a baseline.
Metric.
We measure performance on the GLUE benchmark using a standard metric called the GLUE score, which contains accuracy and correlation scores for each task.
For the sentiment analysis tasks, we measure the accuracy of predicting the sentiment opinion.
For the image classiﬁcation task, we measure the area under the curve (AUC) score.
We run ﬁve different random seeds to report the average results.
The result of an MTL experiment is averaged over the results of all the tasks, unless speciﬁed otherwise.
For the training procedures and other details on the setup, we refer the reader to Appendix C.
3. E XPERIMENTAL R ESULTS
We present use cases of our methods on open-source datasets.
We expected to see improvements via our methods in multi-task and other settings, and indeed we saw such gains across a variety of tasks.
Improving multi-task training.
We apply Algorithm 1 on ﬁve tasks (CoLA, MRPC, QNLI, RTE,
SST-2) from the GLUE benchmark using a state-of-the-art language model BERT LARGE .
We compare the average performance over all ﬁve tasks and ﬁnd that our method outperforms BERT LARGE by 2.35% average GLUE score for the ﬁve tasks.
For the particular setting of training two tasks, our method outperforms BERT LARGE on 7 of the 10 task pairs.
See Figure 5a for the results.
Improving transfer learning.
While our study has focused on multi-task learning, transfer learning is a naturally related goal – and we ﬁnd that our method is also useful in this case.
We validate this by training an LSTM on sentiment analysis.
Figure 5b shows the result with SST being the source task and the rest being the target task.
Algorithm 1 improves accuracy on four tasks by up to 2.5%.
Reweighting training for the same task covariates.
We evaluate Algorithm 2 on the ChestX-ray dataset.
This setting satisﬁes the assumption of Algorithm 2, which requires different tasks to have the same input data.
Across all 14 tasks, we ﬁnd that our reweighting method improves the technique of Kendall et al.
(2018) by 1.3% AUC score.
Compared to training with the unweighted loss, our method improves performance by 5.6% AUC score over all tasks.
3. A BLATION S TUDIES
Model capacity.
We verify our hypothesis that the capacity of the MTL model should not exceed the total capacities of the STL model.
We show this on an LSTM model with the sentiment analysis tasks.
Recall that the capacity of an LSTM model is its output dimension (before the last classiﬁcation layer).
First, we train an MTL model with all tasks and vary the shared module’s capacity to ﬁnd the optimal setting (from 5 to 500).
Then, we train an STL model for each task and ﬁnd the optimal setting similarly.
In Figure 6, we ﬁnd that the performance of MTL peaks when the shared module has capacity 100 .
This is much smaller than the total capacities of all the STL models.
The result −0.
−2.
−0.
−1.
−
[Figure] [Figure]
[Figure]
conﬁrms that constraining the shared module’s capacity is crucial to achieve the ideal performance.
Extended results on CNN/MLP to support our hypothesis are shown in Appendix C.5.
Task covariance.
We apply our metric of task covariance similarity score from Section 2.3 to provide an in-depth study of the covariance alignment method.
The hypothesis is that: (a) aligning the covariances helps, which we have shown in Figure 5a; (b) the similarity score between two tasks increases after applying the alignment.
We verify the hypothesis on the sentiment analysis tasks.
We use the single-task model’s embedding before the LSTM layer to compute the covariance.
First, we measure the similarity score using equation 6 between all six single-task models.
Then, for each task pair, we train an MTL model using Algorithm 1.
We measure the similarity score on the trained MTL model.
Our results conﬁrm the hypothesis (Figure 7): (a) we observe increased accuracy on 13 of 15 task pairs by up to 4.1%; (b) the similarity score increases for all 15 task pairs.
Optimization scheme.
We verify the robustness of Algorithm 2.
After selecting two tasks from the ChestX-ray14 dataset, we test our method by assigning random labels to 20% of the data on one task.
On 20 randomly selected pairs, our method improves over the unweighted scheme by an average 2.4% AUC score and the techniques of Kendall et al.
by an average 0.5% AUC score.
R ELATED W ORK
There has been a large body of recent work on using the multi-task learning approach to train deep neural networks.
Liu et al. (2019a); McCann et al.
(2018) and subsequent follow-up work get stateof-the-art results on the GLUE benchmark, which inspired our study of an abstraction of the MTL model.
Recent work of Zamir et al.
; Standley et al.
(2019) answer which visual tasks to train together via a heuristic which involves intensive computation.
Of particular relevance to this work are those that study the theory of multi-task learning.
The earlier works of Baxter (2000); Ben-David and Schuller (2003) are among the ﬁrst to formally study the importance of task relatedness for learning multiple tasks.
See also the follow-up work of Maurer (2006) which studies generalization bounds of MTL.
A closely related line of work to structural learning is subspace selection, i.e. how to select a common subspace for multiple tasks.
Examples from this line work include Obozinski et al.
; Wang et al.
; Fernando et al.
Evgeniou and Pontil (2004); Micchelli and Pontil (2005) study a formulation that extends support vector machine to the multi-task setting.
See also Argyriou et al.
; Pentina and Ben-David (2015) that provide more reﬁned optimization methods and further study.
The work of Ben-David et al.
(2010) provides theories to measure the differences between source and target tasks for transfer learning in a different model setup.
Recent work of Zhang et al.
(2019) shows adversarially robust methods for domain adaptation.
C ONCLUSIONS AND F UTURE W ORK
In this work, we studied the theory of multi-task learning in linear and ReLU-activated settings.
We veriﬁed our theory and its practical implications through extensive synthetic and real world experiments.
Our work opens up many interesting future questions.
First, could we extend the guarantees for choosing optimization schemes to non-linear settings?
Second, a limitation of our
SVD-based optimization scheduler is that it only applies to settings with the same data.
Could we extend the method for heterogeneous task data?
More broadly, we hope our work inspires further studies to better understand multi-task learning in neural networks and to guide its practice.
A F URTHER R ELATED W ORK
We discuss several lines of studies related to this work.
For complete references, we refer the interested readers to the survey of Ruder (2017) and the surveys on domain adaptation and transfer learning by Pan and Yang (2009); Kouw (2018) for references.
Hard parameter sharing vs soft parameter sharing.
The architecture that we study in this work is also known as the hard parameter sharing architecture.
There is another kind of architecture called soft parameter sharing.
The idea is that each task has its own parameters and modules.
The relationships between these parameters are regularized in order to encourage the parameters to be similar.
Other architectures that have been studied before include the work of Misra et al.
(2016), where the authors explore trainable architectures for convolutional neural networks.
Domain adaptation.
Another closely related line of work is on domain adaptation.
The acute reader may notice the similarity between our study in Section 2.3 and domain adaptation.
The crucial difference here is that we are minimizing the multi-task learning objective, whereas in domain adaptation the objective is typically to minimize the objective on the target task.
See Ben-David et al.
; Zhang et al.
and the references therein for other related work.
Optimization techniques.
Guo et al.
(2019) use ideas from the multi-armed bandit literature to develop a method for weighting each task.
Compared to their method, our SVD-based method is conceptually simpler and requires much less computation.
Kendall et al.
(2018) derive a weighted loss schme by maximizing a Gaussian likelihood function.
Roughly speaking, each task is reweighted by 1 /σ 2 where σ is the standard deviation of the Gaussian and a penalty of log σ is added to the loss.
The values of { σ i } i are also optimized during training.
The exact details can be found in the paper.
The very recent work of Li and Vasconcelos (2019) show empirical results using a similar idea of covariance normalization on imaging tasks for cross-domain transfer.
Shui et al.
(2019) consider multi-task learning from the perspective of adversarial robustness.
Mahmud and Ray (2008) consider using Kolmogorov complexity measure the effectiveness of transfer learning for decision tree methods.
See Mintz et al.
; Misra et al.
; Pentina et al.
; Pentina and Lampert (2017) for other related work.
B M ISSING D ETAILS OF S ECTION
We ﬁll in the missing details left from Section 2.
In Section B.1, we provide rigorous arguments regarding the capacity of the shared module.
In Section B.2, we ﬁll in the details left from Section 2.3, including the proof of Theorem 2 and its extension to the ReLU model.
In Section B.3, we provide the proof of Proposition 3 on the task reweighting schemes.
We ﬁrst describe the notations.
Notations.
We deﬁne the notations to be used later on.
We denote f ( x ) (cid:46) g
( x ) if there exists an absolute constant C such that f ( x ) ≤ Cg ( x ) .
The big-O notation f ( x ) =
O ( g ( x )) means that f ( x ) (cid:46) g ( x ) .
Suppose A ∈ R m × n , then λ max ( A ) denotes its largest singular value and λ min ( A ) denotes its min { m, n } -th largest singular value.
Alternatively, we have λ min ( A ) =
min x : (cid:107) x (cid:107) =
1 (cid:107) Ax (cid:107) .
Let κ ( A ) =
λ max ( A ) /λ min ( A ) denote the condition number of A .
Let Id denotes the identity matrix.
Let U † denote the Moore-Penrose pseudo-inverse of the matrix U .
Let (cid:107) · (cid:107) denote the Euclidean norm for vectors and spectral norm for matrices.
Let (cid:107) · (cid:107) F denote the Frobenius norm of a matrix.
Let (cid:104) A, B, = (cid:105)
Tr( A (cid:62) B ) denote the inner product of two matrices.
The sine function is deﬁne as sin( u, v ) =
(cid:112) 1 − cos( u, v ) 2 , where we assume that sin( u, v ) ≥which is without loss of generality for our study.
B. M ISSING D ETAILS OF S ECTION 2.
We describe the full detail to show that our model setup captures the phenomenon that the shared module should be smaller than the sum of capacities of the single-task models.
We state the following proposition which shows that the quality of the subspace B in equation 1 determines the performance of multi-task learning.
This supplements the result of Proposition 1.
Proposition 4.
In the optimum of f ( · ) (equation 1), each A i selects the vector v within the column span of g B ( X i ) to minimize L ( v, y i ) .
As a corollary, in the linear setting, the optimal B can be achieved at a rotation matrix B (cid:63) ⊆
R d × r by maximizing
[Equation]
Furthermore, any B (cid:63) which contains { θ i } ki =1 in its column subspace is optimal.
In particular, for such a B (cid:63) , there exists { A (cid:63)i } so that B (cid:63) A (cid:63)i
= θ i for all 1 ≤ i ≤ k .
Proof.
Recall the MTL objective in the linear setting from equation 3 as follows:
[Equation]
Note that the linear layer A i can pick any combination within the subspace of B .
Therefore, we could assume without loss of generality that B is a rotation matrix.
i.e. B (cid:62) B = Id .
After ﬁxing B , since objective f ( · ) is linear in A i for all i , by the local optimality condition, we obtain that
[Equation]
Replacing the solution of A i to f ( · ) , we obtain an objective over B .
[Equation]
Next, note that
[Equation]
where we used the fact that A † AA †
=
A † for A =
B (cid:62) X (cid:62)
i X i B in the ﬁrst equation.
Hence we have shown equation 7.
For the ﬁnal claim, as long as B (cid:63) contains { θ i } ki =1 in its column subspace, then there exists A (cid:63)i such that B (cid:63) A (cid:63)i =
θ
i .
The B (cid:63) and { A (cid:63)i } ki =1 are optimal solutions because each θ i is an optimal solution for the single-task problem.
The above result on linear regression suggests the intuition that optimizing an MTL model reduces to optimizing over the span of B .
The intuition can be easily extended to linear classiﬁcation tasks as well as mixtures of regression and classiﬁcation tasks.
Extension to the ReLU setting.
If the shared module’s capacity is larger than the total capacities of the STL models, then we can put all the STL model parameters into the shared module.
As in the linear setting, the ﬁnal output layer A i can pick out the optimal parameter for the i -th task.
This remains an optimal solution to the MTL problem in the ReLU setting.
Furthermore, there is no transfer between any two tasks through the shared module.
B. M ISSING D ETAILS OF S ECTION 2. B.2. T HE E FFECT OF C OSINE S IMILARITY
We consider the effect of varying the cosine similarity between single task models in multi-task learning.
We ﬁrst describe the following proposition to solve the multi-task learning objective when the covariances of the task data are the same.
The idea is similar to the work of Ando and Zhang (2005) and we adapt it here for our study.
Proposition 5.
Consider the reweighted loss of equation 2 with the encoding function being linear, where the weights are { α i } ki =1 .
Suppose the task features of every task have the same covariance:
X (cid:62) i X i =
Σ for all 1 ≤ i ≤ k .
Let Σ = V DV (cid:62) be the singular vector decomposition (SVD) of Σ .
Then the optimum of f ( · ) in equation 3 is achieved at:
[Equation]
where C (cid:63) C (cid:63) (cid:62) is the best rank- r approximation subspace of (cid:80) ki
=
α i U (cid:62) i
y i
y (cid:62)
i U i and X i
=
U i DV (cid:62) is the SVD of X i , for each 1 ≤ i ≤ k .
As a corollary, denote by λ 1 , λ 2 , . . .
, λ k as the singular values of D − 1 V (cid:62) (cid:80) ki
=
α i X (cid:62) i
y i
y (cid:62)
i X i in decreasing order.
Then the difference between an MTL model with hidden dimension r and the all the single task models is bounded by (cid:80) ki
= r +1 λ 2 i .
Proof.
Note that B (cid:63) is obtained by maximizing
[Equation]
Let C = DV (cid:62) B .
Clearly, there is a one to one mapping between B and C .
And we have B =
V D − 1 C .
Hence the above is equivalent to maximizing over
C ⊆
R d × r with k
[Equation]
Note that C ( C (cid:62) C ) − 1 C (cid:62) is a projection matrix onto a subspace of dimension r .
Hence the maximum (denote by C (cid:63) ) is attained at the best rank- r approximation subspace of (cid:80)
[Equation]
To illustrate the above proposition, consider a simple setting where X i is identity for every 1 ≤ i ≤ k , and y i
= e i , i.e. the i -th basis vector.
Note that the optimal solution for the i -th task is ( X (cid:62) i X i ) − 1 X (cid:62)
i y i
= y i .
Hence the optimal solutions are orthogonal to each other for all the tasks, with λ i = 1 for all 1 ≤ i ≤ k .
And the minimum STL error is zero for all tasks.
Consider the MTL model with hidden dimension r .
By Proposition 5, the minimum MTL error is achieved by the best rank- r approximation subspace to (cid:80)
ki
=1 X (cid:62) i
y i
y (cid:62)
i X i =
(cid:80) ki
=
y i y (cid:62) i .
Denote the optimum as B (cid:63)r .
The MTL error is:
[Equation]
Different data covariance.
We provide upper bounds on the quality of MTL solutions for different data covariance, which depend on the relatedness of all the tasks.
The following procedure gives the precise statement.
Consider k regression tasks with data { ( X i , y i ) } ki =1 .
Let θ i =
( X (cid:62) i X i ) † X (cid:62) i
y i denote the optimal solution of each regression task.
Let W ⊆
R
d × k denote the matrix where the i -th column is equal to θ i .
Consider the following procedure for orthogonalizing W for 1 ≤ i ≤ k .
a) Let W (cid:63)i ∈
R
d denote the vector which maximizes (cid:80) ki =1 (cid:104) X i B (cid:107)
X i B (cid:107) , y i (cid:105) 2 over B ∈ R d ; b) Denote by λ j = (cid:80) kj =1 (cid:104)
X j W (cid:63)j (cid:107) X j W (cid:63)j (cid:107) , y j (cid:105) 2 ; c)
For each 1 ≤ i ≤ k , project X i W (cid:63)i off from every column of X i .
Go to Step a).
Proposition 6.
Suppose that r ≤ d .
Let B (cid:63) denote the optimal MTL solution of capacity r in the shared module.
Denote by OPT = (cid:80) ki =1 ( (cid:107) y
i (cid:107) 2 − (cid:107) X
i ( X (cid:62) i X i ) † X
(cid:62)
i y i (cid:107) 2 ) .
Then h ( B (cid:63) ) ≤
OPT − (cid:80) di
= r
+
λ i .
Proof.
It sufﬁces to show that OPT is equal to (cid:80) ki
=1 λ i .
The result then follows since h ( B (cid:63) ) is less than the error given by W (cid:63) 1 , . .
, W (cid:63)k , which is equal to OPT − (cid:80) di
= r
+
λ i .
B.2. P ROOF OF T HEOREM
We ﬁll in the proof of Theorem 2.
First, we restate the result rigorously as follows.
Theorem 2.
For i = 1 , 2 , let ( X i , y i
) ∈
( R m i × d , R m i ) denote two linear regression tasks with parameters θ i ∈ R d .
Suppose that each row of X 1 is drawn independently from a distribution with covariance Σ
1 ⊆ R d × d and bounded l 2 -norm √ L .
Assume that θ (cid:62) 1 Σ 1 θ 1 = 1 w.l.o.g.
Let c ∈ [ κ ( X 2 ) sin( θ 1 , θ 2 ) , 1 / 3] denote the desired error margin.
Denote by ( B (cid:63) , A (cid:63) 1 , A (cid:63) 2 ) the optimal MTL solution.
With probability 1 − δ over the randomness of ( X 1 , y 1 ) , when
[Equation]
we have that (cid:107) B (cid:63) A (cid:63) 2 − θ 2 (cid:107) / (cid:107) θ 2 (cid:107) ≤ 6 c +
[Equation]
1 − 3 c (cid:107) ε 2 (cid:107) / (cid:107) X
2 θ 2 (cid:107) .
We make several remarks to provide more insight on Theorem 2.
• Theorem 2 guarantees positive transfers in MTL, when the source and target models are close and the number of source samples is large.
While the intuition is folklore in MTL, we provide a formal justiﬁcation in the linear and ReLU models to quantify the phenomenon.
• The error bound decreases with c , hence the smaller c is the better.
On the other hand, the required number of data points m 1 increases.
Hence there is a trade-off between accuracy and the amount of data.
• c is assumed to be at most 1 / 3 .
This assumption arises when we deal with the label noise of task 2.
If there is no noise for task 2, then this assumption is not needed.
If there is noise for task 2, this assumption is satisﬁed when sin( θ 1 , θ 2 ) is less than 1 / (3 κ ( X 2 )) .
In synthetic experiments, we observe that the dependence on κ ( X 2 ) and sin( θ 1 , θ 2 ) both arise in the performance of task 2, cf.
Figure 3 and Figure 8, respectively.
The proof of Theorem 2 consists of two steps.
a) We show that the angle between B (cid:63) and θ 1 will be small.
Once this is established, we get a bound on the angle between B (cid:63) and θ 2 via the triangle inequality.
b) We bound the distance between B (cid:63) A 2 and θ 2 .
The distance consists of two parts.
One part comes from B (cid:63) , i.e. the angle between B (cid:63) and θ 2 .
The second part comes from A 2 , i.e. the estimation error of the norm of θ 2 , which involves the signal to noise ratio of task two.
We ﬁrst show the following geometric fact, which will be used later in the proof.
Fact 7.
Let a, b ∈ R d denote two unit vectors.
Suppose that X ∈ R m × d has full column rank with condition number denoted by κ = κ ( X ) .
Then we have
[Equation]
Proof.
Let X =
UDV (cid:62) be the SVD of X .
Since X has full column rank by assumption, we have
X (cid:62) X = XX (cid:62) = Id .
Clearly, we have sin( Xa, Xb )
=
sin( DV (cid:62) a, DV (cid:62) b ) .
Denote by a (cid:48)
=
V (cid:62) a and b (cid:48)
=
V (cid:62) b .
We also have that a (cid:48) b are both unit vectors, sin
) = sin a, b ) .
Let λ 1 , . . .
, λ d denote the singular values of X .
Then,
[Equation]
This concludes the proof.
We ﬁrst show the following Lemma, which bounds the angle between B (cid:63) and θ 2 .
Lemma 8.
In the setting of Theorem 2, with probability 1 − δ over the randomness of task one, we have that
[Equation]
Proof.
We note that h ( B (cid:63) ) ≥ (cid:107) y 1 (cid:107) 2 by the optimality of B (cid:63) .
Furthermore, (cid:104) X 2 B (cid:63)
[Equation]
2 B (cid:63)
[Equation]
2 (cid:105) ≤
[Equation]
y
[Equation]
Hence we obtain that
[Equation]
For the left hand side,
[Equation]
Note that the second term is a chi-squared random variable with expectation σ 21 .
Hence it is bounded by σ 21 (cid:113) log 1 δ with probability at least 1 − δ .
Similarly, the third term is bounded by 2 (cid:107) X 1 θ 1 (cid:107) σ 1 (cid:113) log 1 δ with probability 1 − δ .
Therefore, we obtain the following:
[Equation]
δ
Note that
[Equation]
Therefore,
[Equation]
⇒
[Equation]
, θ 1 )
[Equation] [Equation]
Lemma 7)
By matrix Bernstein inequality (see e.g. Tropp et al. (2015)), when m 1 ≥ 10 (cid:107) Σ 1 (cid:107) log dδ /λ 2min (Σ 1 ) , we have that:
[Equation]
Hence we obtain that κ 2 ( X 1 ) ≤ 3 κ (Σ 1 ) and (cid:107)
X 1 θ 1 (cid:107) 2 ≥ m 1 · θ (cid:62) 1 Σ 1 θ 1 / 2 ≥ m 1 / 2 (where we assumed that θ (cid:62) 1 Σ 1 θ 1 = 1 )
Therefore,
[Equation]
which is at most c 2 /κ 2 ( X 2 ) by our setting of m 1 .
Therefore, the conclusion follows by triangle inequality (noting that both c and sin( θ 1 , θ 2 ) are less than 1 / 2 ).
Based on the above Lemma, we are now to ready to prove Theorem 2.
Proof of Theorem 2.
Note that in the MTL model, after obtaining B (cid:63) , we then solve the linear layer for each task.
For task 2, this gives weight value A (cid:63) 2 :
= (cid:104) X
2 ˆ θ, y 2 (cid:105) / (cid:107)
X
2 ˆ θ (cid:107) 2 .
Thus the regression coefﬁcients for task 2 is B (cid:63) A (cid:63) 2 .
For the rest of the proof, we focus on bounding the distance between
B (cid:63) A (cid:63) 2 and θ 2 .
By triangle inequality,
[Equation]
Note that the second term of equation 8 is equal to
[Equation]
The ﬁrst term of equation 8 is bounded by
[Equation]
Lastly, we have that
[Equation]
By Lemma
[Equation]
we have
[Equation]
Therefore
[Equation]
we conclude that equation 9 is at most
[Equation]
Thus equation 8 is at most the following.
[Equation]
Hence we obtain the desired estimation error of BA (cid:63) 2 .
B.2. E XTENSION TO T HE R E LU M ODEL
In this part, we extend Theorem 2 to the ReLU model.
Note that the problem is reduced to the following objective.
[Equation] [Equation]
We make a crucial assumption that task 1’s input X 1 follows the Gaussian distribution.
Note that making distributional assumptions is necessary because for worst-case inputs, even optimizing a single ReLU function under the squared loss is NP-hard (Manurangsi and Reichman (2018)).
We state our result formally as follows.
Theorem 9.
Let ( X 1 , y
) ∈ ( R m 1 × d , R m 1 ) and ( X 2 , y 2 ) ∈
( R m 2 × d , R m 2 ) denote two tasks.
Suppose that each row of X 1 is drawn from the standard Gaussian distribution.
And y i = a i · ReLU ( X i θ i )
+ ε i are generated via the ReLU model with θ 1 , θ 2 ∈ R d .
Let
E (cid:2) ( a i · ReLU ( X i θ i ))
2 j (cid:3) = 1 for every 1 ≤ j ≤ m 1 without loss of generality, and let σ 21 denote the variance of every of
Suppose that c ≥ sin( θ 1 , θ 2 ) /κ
( X 2 ) .
Denote by ( B (cid:63) , A (cid:63) 1 , A (cid:63) 2 ) the optimal MTL solution of equation 10.
With probability 1 − δ over the randomness of ( X 1 , y 1 ) , when
[Equation]
we have that the estimation error is at most:
[Equation]
Proof.
The proof follows a similar structure to that of Theorem
[Equation]
Without loss of generality
[Equation]
we can assume that θ 1 , θ 2 are both unit vectors.
We ﬁrst bound the angle between B (cid:63) and θ 1 .
By the optimality of B (cid:63)
[Equation]
we have that:
[Equation]
From this we obtain:
[Equation]
Note that each entry of ReLU ( X 1 θ 1 ) is a truncated Gaussian random variable.
By the Hoeffding bound, with probability 1 − δ we have
[Equation]
As for (cid:104) ReLU ( X 1 B (cid:63) ) , ReLU ( X 1 θ 1 ) (cid:105) , we will use an epsilon-net argument over B (cid:63) to show the concentration.
For a ﬁxed B (cid:63) , we note that this is a sum of independent random variables that are all bounded within O (log m 1 δ ) with probability 1 − δ .
Denote by φ the angle between B (cid:63) and θ 1 , a standard geometric fact states that (see e.g. Lemma 1 of Du et al.
for a random Gaussian vector x ∈ R d ,
[Equation]
Therefore, by applying Bernstein’s inequality and union bound, with probability 1 − η we have:
[Equation]
By standard arguments, there exists a set of d O ( d ) unit vectors S such that for any other unit vector u there exists ˆ u ∈ S such that (cid:107) u − ˆ u (cid:107) ≤ min(1 /d 3 , c 2 /κ 2 ( X 2 )) .
By setting η = d − O ( d ) and take union bound over all unit vectors in S , we have that there exists ˆ u ∈ S satisfying (cid:107) B (cid:63) − ˆ u (cid:107) ≤ min(1 /d 3 , c 2 /κ 2 ( X 2 )) and the following:
[Equation]
our setting of m 1 ) where φ (cid:48) is the angle between ˆ u and θ 1 .
Note that
[Equation]
Together we have shown that
[Equation]
Combined with equation 11, by our setting of m 1 , it is not hard to show that
[Equation]
Note that
[Equation]
which implies that sin 2 φ
[Equation]
κ 2 ( X 2 ) (since cos φ (cid:48) 2 ≥ 0 . 9 ).
Finally note that (cid:107) ˆ u − B
[Equation]
(cid:107) ≤ c 2 /κ 2 ( X 2 ) , hence
[Equation]
Overall, we conclude that sin( B (cid:63) , θ 1 ) ≤ O ( c/κ ( X 2 )) .
Hence
[Equation]
For the estimation of a
[Equation]
we have
[Equation]
The ﬁrst part is at most
[Equation]
Similarly, we can show that the second part is at most O ( c ) .
Therefore, the proof is complete.
B. P ROOF OF P ROPOSITION
In this part, we present the proof of Proposition 3.
In fact, we present a more reﬁned result, by showing that all local minima are global minima for the reweighted loss in the linear case.
[Equation]
The key is to reduce the MTL objective f ( · ) to low rank matrix approximation, and apply recent results by Balcan et al.
(2018) which show that there is no spurious local minima for the latter problem .
Lemma 10.
Assume that X (cid:62)
i
X i
=
α i Σ with α i > 0 for all 1 ≤ i ≤ k .
Then all the local minima of f
( A 1 , . . .
, A k ; B ) are global minima of equation 3.
Proof.
We ﬁrst transform the problem from the space of B to the space of C .
Note that this is without loss of generality, since there is a one to one mapping between B and C with C = DV (cid:62) B .
In this case, the corresponding objective becomes the following.
[Equation]
The latter expression is a constant.
Hence it does not affect the optimization solution.
For the former, denote by A ∈ R r × k as stacking the √
α i A
i ’s together column-wise.
Similarly, denote by
Z ∈ R
d
× k as stacking √ α i U (cid:62) i y i together column-wise.
Then minimizing g ( · ) reduces solving low rank matrix approximation: (cid:107) CA − Z (cid:107) 2 F .
By Lemma 3.1 of Balcan et al.
(2018), the only local minima of (cid:107) CA − Z (cid:107) 2 F are the ones where
CA is equal to the best rank- r approximation of Z .
Hence the proof is complete.
Now we are ready to prove Proposition 3.
Proof of Proposition 3.
By Proposition 5, the optimal solution of B (cid:63) for equation 12 is V D −times the best rank- r approximation to α i U (cid:62)
y
i y (cid:62) i
U , where we denote the SVD of X as UDV (cid:62) .
Denote by Q r Q (cid:62) r as the best rank- r approximation to U (cid:62) ZZ (cid:62) U , where we denote by Z = [ √ α 1 y 1 , √ α 2 y 2 , . .
. , √ α k y k ] as stacking the k vectors to a d by k matrix.
Hence the result of
Proposition 5 shows that the optimal solution B (cid:63) is V D − 1 Q r , which is equal to ( X (cid:62) X ) − 1 XQ r .
By Proposition 4, the optimality of B (cid:63) is the same up to transformations on the column space.
Hence the proof is complete.
To show that all local minima are also equal to ( X (cid:62) X ) − 1 XQ r , we can simply apply Lemmaand Proposition 3.
Remark.
This result only applies to the linear model and does not work on ReLU models.
The question of characterizing the optimization landscape in non-linear ReLU models is not well-understood based on the current theoretical understanding of neural networks.
We leave this for future work.
C S UPPLEMENTARY E XPERIMENTAL R ESULTS
We ﬁll in the details left from our experimental section.
In Appendix C.1, we review the datasets used in our experiments.
In Appendix C.2, we describe the models we use on each dataset.
In
Appendix C.3, we describe the training procedures for all experiments.
In Appendix C.4 and Appendix C.5, we show extended synthetic and real world experiments to support our claims.
C. D ATASETS
In this subsection, we describe the synthetic settings we use to verify our theory.
We present more details of the three datasets Sentiment Analysis , General Language Understanding Evaluation (GLUE)
Synthetic settings.
For the synthetic experiments, we draw 10,000 random data samples with dimension d = 100 from the standard Gaussian N (0 , 1) and calculate the corresponding labels based on the model described in experiment.
We split the data samples into training and validation sets with 9,000 and 1,000 samples in each.
For classiﬁcation tasks, we generate the labels by applying a sigmoid function and then thresholding the value to binary labels at 0 .
For ReLU regression tasks, we apply the ReLU activation function on the real-valued labels.
The number of data samples used in the experiments varies depending on the speciﬁcation.
Speciﬁcally, for the task covariance experiment of Figure 3, we ﬁx task 1’s data with m 1 = 9 , 000 training data and vary task 2’s data under three settings: (i) same rotation Q 1 = Q 2 but different singular values D 1 (cid:54) =
D 2 ; (ii) same singular values D
= D 2 but random rotations Q 1 (cid:54) =
Q 2 .
Sentiment analysis.
For the sentiment analysis task, the goal is to understand the sentiment opinions expressed in the text based on the context provided.
This is a popular text classiﬁcation task which is usually formulated as a multi-label classiﬁcation task over different ratings such as positive (+1), negative (-1), or neutral (0).
We use six sentiment analysis benchmarks in our experiments:
• Movie review sentiment (MR): The MR dataset is proposed in Pang and Lee (2005) for detecting positive and negative movie reviews.
In this dataset,
each movie
review consists of a single sentence.
• Sentence subjectivity (SUBJ): The SUBJ dataset is proposed in Pang and Lee (2004) and the goal is to classify whether a given sentence is subjective or objective.
• Customer reviews polarity (CR): The CR dataset, collected by Hu and Liu (2004), provides customer reviews of various products.
The goal of this task is to categorize positive and negative reviews.
• Question type (TREC):
The TREC dataset is collected by Li and Roth (2002).
The aim is to classify a question into 6 question types.
• Opinion polarity (MPQA): The MPQA dataset detects whether an opinion is polarized or not (Wiebe et al.
• Stanford sentiment treebank (SST): The SST dataset, created by Socher et al. (2013), is an extension of the MR dataset.
The General Language Understanding Evaluation (GLUE) benchmark.
GLUE is a collection of natural language understanding tasks including question answering, sentiment analysis, text similarity and textual entailment problems.
The GLUE benchmark is a state-of-the-art multi-task learning benchmark for both academia and industry.
We select ﬁve representative tasks including
CoLA, MRPC, QNLI, RTE, and SST-2 to validate our proposed method.
We emphasize that the goal of this work is not to come up with a state-of-the-art result but rather to provide insights into the working of multi-task learning.
It is conceivable that our results can be extended to the entire dataset as well.
This is left for future work.
More details about the GLUE benchmark can be found in the original paper (Wang et al.
(2018a)).
ChestX-ray14.
The ChestX-ray14 dataset (Wang et al.
(2017)) is the largest publicly available chest X-ray dataset.
It contains 112,120 frontal-view X-ray images of 30,805 unique patients.
Each image contains up to 14 different thoracic pathology labels using automatic extraction methods on radiology reports.
This can be formulated as a 14-task multi-label image classiﬁcation problem, where each task is a binary classiﬁcation problem.
The ChestX-ray14 dataset is a representative dataset in the medical imaging domain as well as in computer vision.
We use this dataset to examine our proposed task reweighting scheme since it satisﬁes the assumption that all tasks have the same input data but different labels.
C. M ODELS
We describe the models we use for the experiments.
Synthetic settings.
For the synthetic experiments, we use the linear regression model, the logistic regression model and a one-layer neural network with the ReLU activation function.
Sentiment analysis.
For the sentiment analysis experiments, we consider three different models including multi-layer perceptron (MLP), LSTM, CNN:
• For the MLP model, we average the word embeddings of a sentence and feed the result into a two layer perceptron, followed by a classiﬁcation layer.
• For the LSTM model, we use the standard one-layer single direction LSTM as proposed by Lei et al.
(2018), followed by a classiﬁcation layer.
• For the CNN model, we use the model proposed by Kim (2014) which uses one convolutional layer with multiple ﬁlters, followed by a ReLU layer, max-pooling layer, and classiﬁcation layer.
We follow the protocol of Kim (2014) and set the ﬁlter window size to be { 3 , 4 , 5 } .
We use the pre-trained GLoVe embeddings trained on Wikipedia 2014 and Gigaword 5 corpora 2 .
We ﬁne-tune the entire model in our experiments.
In the multi-task learning setting, the shared modules include the embedding layer and the feature extraction layer (i.e. the MLP, LSTM, or CNN model).
Each task has its separate output module.
GLUE.
For the experiments on the GLUE benchmark, we use a state-of-the-art language model called BERT (Devlin et al.
For each task, we add a classiﬁcation/regression layer on top it as our model.
For all the experiments, we use the BERT LARGE uncased model, which is a 24 layer network as described in Devlin et al.
For the multi-task learning setting, we follow the work of Liu et al.
(2019a)
and use BERT LARGE as the shared module.
ChestX-ray14.
For the experiments on the ChestX-ray14 dataset, we use the DenseNet model proposed by Rajpurkar et al.
(2017) as the shared module, which is a 121 layer network.
For each task, we use a separate classiﬁcation output layer.
We use the pre-trained model 3 in our experiments.
C. T RAINING P ROCEDURES
In this subsection, we describe the training procedures for our experiments.
M T L pe r f o r m an c
e
o v e r S T L ( S pea r m an c
o rr e l a t i on )
[Figure]
r f o
[Figure]
c e o v e r S
[Figure]
L (
[Figure]
u r a
[Figure]
Synthetic settings.
For the synthetic experiments, we do a grid search over the learning rate from { 1 e − 4 , 1 e − 3 , 1 e − 2 , 1 e − 1 } and the number of epochs from { 10 , 20 , 30 , 40 , 50 } .
We pick the best results for all the experiments.
We choose the learning rate to be 1 e − 3 , the number of epochs to be 30 , and the batch size to be 50.
For regression task, we report the Spearman’s correlation score
For classiﬁcation task, we report the classiﬁcation accuracy.
Sentiment analysis.
For the sentiment analysis experiments, we randomly split the data into training, dev and test sets with percentages 80% , 10% , and 10% respectively.
We follow the protocol of Lei et al.
(2018) to set up our model for the sentiment analysis experiments.
The default hidden dimension of the model (e.g. LSTM) is set to be 200, but we vary this parameter for the model capacity experiments.
We report the accuracy score on the test set as the performance metric.
GLUE.
For the GLUE experiments, the training procedure is used on the alignment modules and the output modules.
Due to the complexity of the BERT LARGE module, which involves 24 layers of non-linear transformations, we ﬁx the BERT LARGE module during the training process to examine the effect of adding the alignment modules to the training process.
In general, even after ﬁne-tuning the BERT LARGE module on a set of tasks, it is always possible to add our alignment modules and apply Algorithm 1.
For the training parameters, we apply grid search to tune the learning rate from { 2e − 5 , 3e − 5 , 1e − 5 } and the number of epochs from { 2 , 3 , 5 , 10 } .
We choose the learning rate to be 2e − 5 , the number of epochs to be 5 , and with batch size 16 for all the experiments.
We use the GLUE evaluation metric (cf.
Wang et al.
(2018b))
and report the scores on the development set as the performance metric.
ChestX-ray14.
For the ChestX-ray14 experiments, we use the conﬁguration suggested by Rajpurkar et al.
and report the AUC score on the test set after ﬁne-tuning the model for 20 epochs.
C. E XTENDED S YNTHETIC E XPERIMENTS
Varying cosine similarity on linear and ReLU models.
We demonstrate the effect of cosine similarity in synthetic settings for both regression and classiﬁcation tasks.
Synthetic tasks.
We start with linear settings.
We generate 20 synthetic task datasets (either for regression tasks, or classiﬁcation tasks) based on data generation procedure and vary the task similarity between task 1 and task i .
We run the experiment with a different dataset pairs (dataset 1 and dataset i ).
We compare the performance gap between MTL and STL model.
Results.
From Figure 8a and Figure 8a, we ﬁnd that for both regression and classiﬁcation settings, with the larger task similarity the MTL outperforms more than STL model and the negative transfer could occur if the task similarity is too small.
[Figure] [Figure]
ReLU settings.
We also consider a ReLU-activated model.
We use the same setup as the linear setting
[Figure]
but apply a ReLU activation to generate the data.
Similar results are shown in Figure 8c, 8d.
Higher rank regimes for ReLU settings.
We provide further validation of our results on ReLUactivated models.
Synthetic tasks.
In this synthetic experiment, there are two sets of model parameters Θ 1 ⊆
R
d × r and Θ 2 ⊆
R d × r ( d = 100 and r = 10 ).
Θ 1 is a ﬁxed random rotation matrix and there are m 1 =data points for task 1 .
Task 2 ’s model parameter is Θ 2 = α Θ 1 + (1 − α )Θ (cid:48) , where Θ (cid:48) is also a ﬁxed rotation matrix that is orthogonal to Θ 1 .
Note that α is the cosine value/similarity of the principal angle between Θ 1 and Θ 2 .
We then generate X 1 ⊆
R m 1 × d and X 2 ⊆
R m 2 ×
d from Gaussian.
For each task, the labels are y i
= ReLU ( X i Θ i )
e + ε i , where e ∈ R r is the all ones vector and ε i is a random Gaussian noise.
Given the two tasks, we use MTL with ReLU activations and capacity H = 10 to co-train the two tasks.
The goal is to see how different levels of α or similarity affects the transfer from task two to task one.
Note that this setting parallels the ReLU setting of Theorem 9 but applies to rank r = 5 .
Results.
In Figure 9 we show that the data size, the cosine similarity between the STL solutions and the alignment of covariances continue to affect the rate of transfer in the new settings.
The study shows that our conceptual results are applicable to a wide range of settings.
Evaluating Algorithm 1 on linear and ReLU-activated models.
We consider the synthetic example in Section 2.3 to compare Algorithm 1 and the baseline MTL training.
Recall that in the example, when the source and target tasks have different covariance matrices, MTL causes negative transfer on the target task.
Our hypothesis in this experiment is to show that Algorithm 1 can correct the misalignment and the negative transfer.
Synthetic tasks.
We evaluate on both linear and ReLU regression tasks.
The linear case follows the example in Section 2.3.
For the ReLU case, the data is generated according to the previous example.
Results.
Figure 10 conﬁrms the hypothesis.
We observe that Algorithm 1 corrects the negative transfer in the regime where the source task only has limited amount of data.
Furthermore, Algorithmmatches the baseline MTL training when the source task has sufﬁciently many data points.
C. E XTENDED A BLATION S TUDIES
Cross validation for choosing model capacities.
We provide an cross validation experiment to indicate how we choose the best performing model capacities in Figure 6.
This is done on the six sentiment analysis tasks trained with an LSTM layer.
In Figure 11, we vary the model capacities to plot the validation accuracies of the MTL model trained with all six tasks and the STL model for each task.
The result complements Figure 6 in Section 3.3.
[Figure]
Choosing model capacities for CNN and MLP.
Next we verify our result on model capacities for
CNN and MLP models.
We select the SST and MR datasets from the sentiment analysis tasks for this experiment.
We train all three models CNN, MLP and LSTM by varying the capacities.
Results.
From Figure 12 we observe that the best performing MTL model capacity is less than total best performing model capacities of STL model on all models.
[Figure]
The effect of label noise on Algorithm 2.
To evaluate the robustness of Algorithm 2 in the presence of label noise, we conduct the following experiment.
First, we select two tasks from the ChestXray14 dataset.
Then, we randomly pick one task to add 20% of noise to its labels by randomly ﬂipping them with probability 0 .
We compare the performance of training both tasks using our reweighting scheme (Algorithm 2) vs. the reweighting techniques of Kendall et al.
and the unweighted loss scheme.
Results.
On 20 randomly chosen task pairs, our method improves over the unweighted training scheme by 2.4% AUC score and 0.5% AUC score over Kendall et al.
(2018) averaged over thetask pairs.
Figure 13 shows 5 example task pairs from our evaluation.
[Figure]