F RUSTRATINGLY EASY QUASI - MULTITASK LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose the technique of quasi-multitask learning (Q-MTL), a simple and easy to implement modiﬁcation of standard multitask learning, in which the tasks to be modeled are identical.
We illustrate it through a series of sequence labeling experiments over a diverse set of languages, that applying Q-MTL consistently increases the generalization ability of the applied models.
The proposed architecture can be regarded as a new regularization technique encouraging the model to develop an internal representation of the problem at hand that is beneﬁcial to multiple output units of the classiﬁer at the same time.
This property hampers the convergence to such internal representations which are highly speciﬁc and tailored for a classiﬁer with a particular set of parameters.
Our experiments corroborate that by relying on the proposed algorithm, we can approximate the quality of an ensemble of classiﬁers at a fraction of computational resources required.
Additionally, our results suggest that Q-MTL handles the presence of noisy training labels better than ensembles
I NTRODUCTION
Ensemble methods are frequently used in machine learning applications due to their tendency of increasing model performance.
While the increase in the prediction performance is undoubtedly an important aspect when we train a model, it should not be forgotten that the increased performance of ensembling comes at the price of training multiple models for solving the same task.
The question that we tackle in this paper is the following: Can we enjoy the beneﬁts of ensemble learning, while avoiding its overhead for training models from scratch multiple times?
This question is highly relevant these days, since state-of-the-art neural models tend to be extremely resourceintensive on their own (Strubell et al., 2019), prohibiting their inclusion in a traditional ensemble setting.
Our proposed architecture simultaneously offers the beneﬁt of ensemble learning, while avoiding its drawback of training multiple models.
The method introduced here employs a special form of multitask learning (MTL).
Caruana (1997) argues in his seminal work that MTL can be a useful source of introducing inductive bias into machine learning models.
Standard MTL have been shown to be fruitfully applicable in solving a series of NLP tasks:
Collobert & Weston (2008); Plank et al.
; Rei (2017); Kiperwasser & Ballesteros (2018); Sanh et al.
(2018), inter alia .
We introduce quasi-multitask learning (Q-MTL), where the goal is to simultaneously learn multiple neural models that solve identical tasks , while relying on a shared representation layer.
Besides the considerable speedup that comes with the proposed technique, we additionally argue that by applying multiple output units on top of a shared parameter set is beneﬁcial, as we can avoid converging to such degenerate internal representations that are highly tailored for a particular classiﬁcation model.
In that sense, Q-MTL can also be viewed as an implicit regularizer, which prevents neural networks to develop such an internal representation which is not generic enough to provide useful input to multiple classiﬁcation units simultaneously.
Our experiments with Q-MTL illustrate that the presence of multiple classiﬁer layers for the same task affect each other positively – similar to ensemble learning – without the additional overhead of actually training multiple models.
A similar technique have already been derived from MTL called Pseudo-Task Augmentation (Meyerson & Miikkulainen, 2018), which builds on the idea of common representation, but the manage-
Verb Noun
Noun
ReLU co ﬀ ee
BiLSTM
ReLU co ﬀ ee
BiLSTM
ReLU co ﬀ ee
BiLSTM (a) Sequence of STLs
BiLSTM co ﬀ ee
Verb Noun
Noun Σ
Noun
ReLU ReLU
ReLU (b) Q-MTL
Verb Noun
Noun Σ
Noun
ReLU co ﬀ ee
BiLSTM
ReLU co ﬀ ee
BiLSTM
ReLU co ﬀ ee
BiLSTM (c) Ensemble
A PPLIED MODELS
We release all our source code used for our experiments at anonymized .
Our models are based on the sequence classiﬁcation framework from Plank et al.
(2016) implemented in DyNet (Neubig et al., 2017).
Figure 1 provides a visual summary of the different architectures we implemented.
Figure 1b highlights that Q-MTL has the beneﬁt of training multiple classiﬁcation models over the same internal representation, as opposed to traditional ensemble model, which requires the training of multiple LSTM parameters as well (cf.
Figure 1c).
2. B ASELINE ARCHITECTURE
Our baseline classiﬁer is a bidirectional LSTM (Hochreiter & Schmidhuber, 1997) incorporating character and word level embeddings.
We ﬁrst compute the input embedding for the network at position i as
[Equation]
where ⊕ is the concatenation operator, w i denotes the word embedding, −→ c i and ←− c i refers to the left-to-right and right-to-left character-based embeddings, respectively.
We subsequently feed e i into a bi-LSTM, which determines a hidden representation h i ∈
R m for every token position as h i = −→ h i ⊕ ←− h
i , i.e., the concatenation of the hidden states of the two LSTMs processing the input from its beginning to the end, and in reverse direction.
The ﬁnal output of the network for token position i gets computed as
[Equation]
with V ∈ R h × m and b V ∈ R h denoting the weight matrix and the bias of a perceptron unit, whereas W ∈ R h × c and b W ∈ R c are the parameters of the neuron performing classiﬁcation over the c target classes.
2. Q-MTL ARCHITECTURE
The Q-MTL network behaves similarly to the model introduced in Section 2.1, with the notable exception that it trains k distinct classiﬁcation models, all of which operate over the same hidden representation as input obtained from a single bi-LSTM unit.
More concretely, we replace the single prediction of the STL model from Eq.
1 by a series of predictions for Q-MTL according to
[Equation]
with j ∈ { 1 , . . . , k } .
As argued before, this approach behaves favorably from a computational point of view, as it relies on a shared representation
h
i for all the k classiﬁcation units.
The loss of the network for token position i and gold standard class label y ∗ i can be conveniently generalized as
[Equation]
where CE denotes categorical cross entropy loss and k is the number of (identical) tasks in the
Q-MTL model, with the special case of k = 1 resulting in standard single task learning (STL).
Losses from the different outputs can be aggregated efﬁciently during backpropagation, hence the shared LSTM cell beneﬁt from multiple error signals without the actual need of going through multiple individual forward and backward passes.
Q-MTL outputs k predictions by all of its prediction units, however, we can as well derive a combined prediction from the distinct outputs of Q-MTL according to
[Equation]
which essentially is a weighted average according to the predicted probabilities of the distinct models.
As introducing averaging at the model-level would eliminate diversity of the individual classiﬁers (Lee et al., 2015), this kind of averaging took place in a post-hoc manner, only when making predictions.
2. T RADITIONAL ENSEMBLE MODEL
As an additional model, we also employ a traditional ensemble of k independently trained STL models.
We deﬁne the prediction of the ensemble model by averaging the predictions of k independent models as
[Equation]
The distinctive difference between Eq. 4 and the Q-MTL model formulation in Eq.
3 is that ensembling relies on the hidden representations originating from k independently trained LSTM models as denoted by the superscripts of the hidden states in h ( j ) i .
Such an ensemble necessarily requires approximately k -times as much computational resources compared to Q-MTL, due to the LSTM models being trained in total isolation.
For the above reason, ensembling is a strictly more expensive form of training a model, for which reason we regard its performance as a glass ceiling for
Q-MTL.
E XPERIMENTS
Our model uses character embeddings of 100 dimensions and the word representations get initialized by the 64-dimensional pre-trained polyglot word embeddings (Al-Rfou et al., 2013) as suggested by
Plank & Agi´c (2018).
We use a one-layered bi-LSTM which outputs hidden vectors h i ∈ R 200 as a concatenation of −→ h
i , ←− h i ∈ R 100 .
Instead of directly applying a fully-connected layer to perform classiﬁcation based on h i , we ﬁrst transform h i by an intermediate perceptron unit with ReLU [Table] [Table] [Table]
activation. The perceptron transforms h i into 20 dimensions, that is, we have V ∈ R 20 × 200 .
Our motivation with the extra non-linearity introduced by ReLU is to encourage an increased diversity in the behavior of the different output units.
Upon training the LSTMs, we used the default architectural settings employed by Plank et al.
(2016), i.e., we relied on a word dropout rate of 0.25 (Kiperwasser & Goldberg, 2016) and an additive
Gaussian noise (with σ = 0 . 2 ) over the input embeddings.
We trained all our models for 20 epochs using stochastic gradient descent with a batch size of 1 .
First, we assess the quality of Q-MTL towards POS tagging, then we evaluate it on named entity recognition as well.
When comparing the performance of different approaches, Q-MTL models are compared against the average performance of k STL models, where k denotes the number of task in the case of Q-MTL.
The k STL models are also used to derive a single prediction by the ensemble model.
3. POS TAGGING EXPERIMENTS
We set our POS tagging related experiments on 10 treebanks from the Universal Dependencies dataset v2.2 (Nivre et al., 2018), namely the Greek-GDT (el), English-LinES (en), Basque-BDT (eu), Finnish-FTB (ﬁ), Croatian-SET (hr), Hungarian-Szeged (hu), Indonesian-GSD (id), DutchAlpino (nl), Tamil-TTB (ta) and Turkish-IMST (tr) treebanks.
These treebanks not only cover a typologically diverse set of languages, but they also vary substantially in the number of available training sequences, as illustrated in Table 1.
Table 1 also illustrates the typological diversity of the investigated languages, as the average number of occurrences per distinct word forms vary substantially, i.e., between 2.4 for Tamil and 6.9 for Dutch.
3.1. E XPERIMENTS WITH THE NUMBER OF TASKS
We ﬁrst investigate how does changing the value for k , i.e., the number of simultaneously learned tasks, affects the performance of Q-MTL.
We experimented with k ∈ { 1 , 10 , 30 } .
Based on the results in Table 2, we set the number of tasks to be employed as k = 10 for all upcoming experiments.
In order to choose k without overﬁtting to the training data, this experiment was conducted on the development set.
3.1. C OMPARING Q-MTL WITH STL
Following the recommendation in (Dodge et al., 2019), we report learning curves over the development set as a function of the number epochs in Figure
As a general observation, we can see that
Q-MTL tends to perform consistently better than STL models right from the beginning of training.
Directly comparing the classiﬁers One beneﬁt of Q-MTL is that it learns k different classiﬁcation models during training with only a marginal computational overhead compared to training a
STL baseline, since all the tasks share a common internal representation.
As discussed earlier, we can combine the predictions from the k classiﬁers from Q-MTL according to Eq.
It is also possible, however, to use the k distinct predictions of Q-MTL.
In what follows next, we compare the [Figure] performance of the k STL models we train to the k classiﬁers that are incorporated within a Q-MTL model.
Upon comparing the performance of a Q-MTL classiﬁer with a STL model
[Figure]
we made it sure that the overlapping parameters (matrices V and W ) were initialized with the same values and that they receive the training instances in the exact same order.
This way the performance achieved by the i th output of Q-MTL is directly comparable with the i th STL baseline.
Comparison of the results of the individual outputs of Q-MTL and their corresponding STL counterpart are included in Figure
[Figure]
Training Q-MTL models with k tasks simultaneously is not only faster than training k distinct STL models separately, but the individual Q-MTL models typically outperform their baseline counterparts evaluated against both the development and the test data.
The regularizing effect of Q-MTL We have argued earlier that Q-MTL has an implicit regularizing effect.
Among most recent techniques, such as dropout (Srivastava et al., 2014), weight decay (Krogh & Hertz, 1992) is one of the most typical form of regularization for fostering the generalization capability of the learned models.
When employing weight decay, we add an extra term penalizing the magnitude of the values learned by our model, which results in an overall shrinkage in the values of the model parameters.
F r o b e n i u s n o r m el en eu fi hr
STL
Q-MTL
F r o b e n i u s n o r m hu
STL
Q-MTL id
STL
Q-MTL nl
STL
Q-MTL
ta
STL
Q-MTL tr
V
W
Figure 4 illustrates that the effects of employing Q-MTL is similar to applying weight decay, as the
Frobenius norm of the parameter matrices from the classiﬁers of Q-MTL are substantially smaller than those of the STL classiﬁers.
This observation holds for both the of parameter sets V and W .
Recall that the initial values for these matrices were identical for both Q-MTL and STL.
3.1. C OMPARISON TO AN ENSEMBLE OF CLASSIFIERS
We have provided a detailed comparison of the STL and Q-MTL models so far.
We next extend their comparative evaluation with the ensemble model.
Upon providing a comparison for the different approaches, we also assess their sensitivity towards the presence of noisily labeled tokens during training.
To do so, we conducted multiple experiments for each language, for which we randomly replaced the true class label of a token by some predeﬁned probability p ∈ { 0 , 0 . 1 , 0 . 2 , 0 . 3 } .
During the random replacement of the class labels, we ensured that the same tokens got randomly relabeled by the same label for the different approaches.
Figure 5 contains the performance of the three different models in conjunction with the different amounts of noisy labels introduced to the training set.
We can observe from Figure 5 that Q-MTL outperforms STL for all the languages irrespective to the amount of noisy tokens being present encountered during training.
Figure 5 further reveals that the performances of the ensemble models – which are based on the predictions of the STL classiﬁers – are dominantly better than the average performance of the individual STL models.
When mislabeled tokens are not present in the training data at all, ensemble also has a slight advantage over Q-MLT, however, this advantage of the ensembling model gradually fades out as the proportion of noisy training labels increases.
Indeed, for the case when 30% of the training labels are randomly replaced, the performance of Q-MTL reaches that of the ensemble model (cf.
the rightmost subplot in Figure 5).
The Q-MTL approach has the additional beneﬁt over the ensemble model that it requires a fraction of computational resources as we will demonstrate it in Section 3.2.
As a ﬁnal interesting note, the Q-MTL has an improved performance for Indonesian as the amount of noisy training labels increases.
A possible explanation for this is that corrupting the class labels of the training data can be viewed as an alternative form of label smoothing (Szegedy et al., 2016), which is known to increase the generalization ability of neural models.
3. C OMPARISON OF TRAINING TIMES
One of the main beneﬁts of Q-MTL resides in its training efﬁciency compared to traditional ensemble models as also demonstrated by Figure 6, which includes the training times for the different approaches.
We plot the training times on the logarithmic scale for better readability for both k = and k = 30 .
We can see that the training times for STL and Q-MTL practically concur, whereas the overall costs of ensembling exceeds the training time of STL and Q-MTL models by a factor of k .
The training times reported in Figure 6 were obtained without GPU acceleration – on an Intel Xeon
E7-4820 CPU – in order to simulate a setting with limited computational resources.
We also repeated training on a TITAN Xp GPU.
The GPU-based training was 3 to 10 times quicker depending on the languages, but the relative performance between the different approaches remained the same, i.e.,
STL and Q-MTL training times did not differ substantially, whereas the ensemble model took ktimes as much time to be created.
3. E VALUATION ON N AMED E NTITY R ECOGNITION
We also conducted experiments on the CoNLL 2002/2003 shared task data on named entity recognition (NER) in English, Spanish and Dutch (Tjong Kim Sang, 2002;
Tjong Kim Sang & De Meulder, 2003).
For these experiments, we report performance in terms of overall F1 scores calculated by the ofﬁcial scorer of the shared task.
We trained models with k = 10 and compared the average performance of the individual STL models to the performance of the Q-MTL and ensemble models.
Table 4a shows the results for NER over the different languages, corroborating our previous observation that Q-MTL is capable of closing the gap between the performance of STL models and the much more resource-intensive ensemble model derived from k independent models.
In our POS tagging experiment, we trained models on treebanks of radically differing sizes (cf.
Table 1), whereas during our NER experiments, we had access to training data sets of comparable sizes (ranging between 218K and 273K tokens).
In order to simulate the effects of having access to limited training data on NER as well, we artiﬁcially relied on only 10% of the available training sets.
These results for the limited training data setting are included in Table 4b, from which we can see that Q-MTL manages to preserve a larger fraction of its original performance, i [Figure] e., 87.5% on average as opposed to the ensemble and STL models, which preserved only 86.7% and 86.4% of their original F-scores, respectively
[Figure]
C OMPARISON TO P SEUDO -T ASK A UGMENTATION
An achitecture introduced by Meyerson & Miikkulainen (2018) called Pseudo-Task Augmentation (PTA) uses a similar architecture to leverage a better representation of the task by ﬁtting multiple outputs to the same task.
This architecture makes a series of predictions according to
[Equation]
[Figure]
[Figure]
l
[Figure]
s
[Figure]
t
i
[Figure]
k
[Figure]
method
STL
Q
[Figure]
Ensemble
[Table]
[Figure]
[Table]
[Figure]
The main difference between PTA and Q-MTL is that Q-MTL uses an extra transformation and a
ReLU non-linearity over the hidden representation of the LSTM (cf.
Eq.
2 and Eq.
5 for Q-MTL and PTA, respectively).
Another key difference is that PTA uses model selection (MS), whereas Q-MTL relies on model averaging (MA).
This means that PTA makes prediction for test instances during inference based on the model which achieves best performing dev set accuracy.
Q-MTL, however, aggregates all the models according to 3.
{+As Figure 7 shows the different settings for model selection and Multi-Layered Perceptron (MLP) that explore the methods used in PTA (MS @ 
+}
MLP) to Q-MTL (MA @ 20 MLP).
We also we evaluated the different combinations of model selection/averaging and the usage of additional nonlinearity for completeness.
In these experiments 20 MLP means the MLP has a 20 dimensional
[Figure]
output, while 0 MLP means there is no MLP layer added to the tasks.
Note, that by removing the
MLP layer we also remove the ReLU activation.
Figure 7 demonstrates that the Q-MTL model with its MLP layer can facilitate the use of model averaging shown in Eq.
3 as it outperforms the Q-MTL using model selection (MS @ 20 MLP).
On the other hand in the case no MLP is used (0 MLP) with which it is indeed discouraged to ensemble the models, we can see that MS often outperforms MA in case of 0 MLP.
Additionally, we can see that the MLP layer improves the tolerance of the models to the increasing label noise, as it outperforms 7 out of 10 treebanks the model not employing extra non-linearity.
Interestingly when the train set contains high label noise, the ensemble of linear classiﬁers outperforms the model selected ones.
R ELATED WORK
Caruana (1997) showed that neural networks can be trained for multiple tasks, leveraging cross domain information.
More recently, Søgaard & Goldberg (2016); Sanh et al.
(2018) argues that solving low-level NLP tasks can improve the performance of high level tasks.
Additionally, Plank et al.
; Bingel & Søgaard (2017) show that better performing models can be trained by introducing multiple auxiliary tasks.
Rei (2017) proposes an auxiliary task for NLP sequence labeling tasks, where the auxiliary tasks is to predict the previous and next word in the sequence.
Our results complement these ﬁndings by showing that better generalization can also be achieved if we learn multiple models for the same task concurrently.
Meyerson & Miikkulainen (2018) introduced Pseudo-Task Augmentation a similar architecture that aims to build a robust internal representation from multiple classiﬁer units that optimize for the same task in the same neural network.
Their work contains policies that simulate linear independence (refered as: perturb, hyperturb, independent dropout) in the set of linearly dependent classiﬁer units, while making use of the dependence in other cases to improve accuracy (refered as: greedy).
Section 4 describes the similarities and differences to our method.
PTA architecture is evaluated on multitask as well, while our work only considers single tasks at the moment.
Ruder & Plank (2018) has shown that self-learning and tri-training can be adapted to deep neural nets in the semi-supervised regime.
Their tri-training architecture resembles our approach in that they were utilizing multiple classiﬁer units that were built on top of a common representation layer for providing labels to previously unlabeled data.
Cross-view training (CVT) (Clark et al., 2018) resembles Q-MTL in that it also employs a shared biLSTM layer used by multiple output layers.
The main difference between CVT and Q-MTL is that we are utilizing an bi-LSTM to solve the same task multiple times in a supervised setting, whereas
Clark et al.
(2018) used it to solve different tasks in a semi-supervised scenario.
A series of studies have made use of ensemble learning in the context of deep learning (Hansen & Salamon, 1990; Krogh & Vedelsby, 1995; Lee et al., 2015; Huang et al., 2017).
Our proposed model is also related to the line of research on mixture of experts proposed by Jacobs et al.
(1991), which has already been applied successfully in NLP before (Le et al., 2016).
The main difference in our proposed architecture is that the internal LSTM representation that we train are shared across the classiﬁers, hence a more efﬁcient training could be achieved as opposed to training multiple independent expert models as it was done in (Shazeer et al., 2017).
Model distillation (Hinton et al., 2015) is an alternative approach for making computationally demanding models more effective during inference, however, the approach still requires training of a “cumbersome” model ﬁrst.
C ONCLUSIONS
We proposed quasi-multitask learning (Q-MTL), which can be viewed as an efﬁciently trainable alternative of traditional ensembles.
We additionally demonstrated that it acts as an implicit form of regularization as well.
In our experiments, Q-MTL consistently outperformed the single task learning (STL) baseline for both POS tagging and NER.
We have also illustrated that Q-MTL generalizes better on smaller and noisy datasets compared to both STL and ensemble models.
The computational overhead for the additional classiﬁcation units in Q-MTL is inﬁnitesimal due to the effective aggregation of the losses and the shared recurrent unit between the identical tasks.
Although we evaluated our approach over sequence classiﬁcation tasks, the general idea can be applied for other network architectures and beyond NLP applications as well