Quasi-Multitask Learning: an Efficient Surrogate for Obtaining Model Ensembles (2020)

Abstract: We propose the technique of quasi-multitask learning (Q-MTL), a simple and easy to implement modification of standard multitask learning, in which the tasks to be modeled are identical. With this easy modification of a standard neural classifier we can get benefits similar to an ensemble of classifiers with a fraction of the resources required. We illustrate it through a series of sequence labeling experiments over a diverse set of languages, that applying Q-MTL consistently increases the generalization ability of the applied models. The proposed architecture can be regarded as a new regularization technique that encourages the model to develop an internal representation of the problem at hand which is beneficial to multiple output units of the classifier at the same time. Our experiments corroborate that by relying on the proposed algorithm, we can approximate the quality of an ensemble of classifiers at a fraction of computational resources required. Additionally, our results suggest that Q-MTL handles the presence of noisy training labels better than ensembles.

## introduction ensemble methods are frequently used in machine learning applications due to their tendency of increasing model performance. while the increase in the prediction performance is undoubtedly an important aspect when we train a model, it should not be forgotten that the increased performance of ensembling comes at the price of training multiple models for solving the same task. the question that we tackle in this paper is the following: can we enjoy the benefits of ensemble learning, while avoiding its overhead for training models from scratch multiple times? this question is highly relevant these days, since state-of-theart neural models tend to be extremely resourceintensive on their own @xcite , pro-hibiting their inclusion in a traditional ensemble setting. our proposed architecture simultaneously offers the benefit of ensemble learning, while avoiding its drawback of training multiple models. the method introduced here employs a special form of multitask learning (mtl). caruana @xcite argues in his seminal work that mtl can be a useful source of introducing inductive bias into machine learning models. standard mtl have been shown to be fruitfully applicable in solving a series of nlp tasks: @xcite ; @xcite ; @xcite ; @xcite ; @xcite , inter alia. we introduce quasi-multitask learning (q-mtl), where the goal is to simultaneously learn multiple neural models that solve identical tasks, while relying on a shared representation layer. besides the considerable speedup that comes with the proposed technique, we additionally argue that by applying multiple output units on top of a shared parameter set is beneficial, as we can avoid converging to such degenerate internal representations that are highly tailored for a particular classification model. in that sense, q-mtl can also be viewed as an implicit regularizer. our experiments with q-mtl illustrate that the presence of multiple classifier layers for the same task affect each other positively -similar to ensemble learning -without the additional overhead of actually training multiple models. a similar technique have already been derived from mtl called pseudo-task augmentation @xcite , which builds on the idea of common representation, but the management of these tasks differs. we conducted experiments comparing the two methods for a greater comprehension of the differences. ## applied models we release all our source code used for our experiments at https://github.com/n0rbi/ quasi-multitask-learning/ . our models are based on the sequence classification framework from @xcite implemented in dynet @xcite . figure 1 provides a visual summary of the different architectures we implemented. figure 1b highlights that q-mtl has the benefit of training multiple classification models over the same internal representation, as opposed to traditional ensemble model, which requires the training of multiple lstm parameters as well (cf. figure 1c ). ## experiments our model uses character embeddings of 100 dimensions and the word representations get initialized by the 64-dimensional pre-trained polyglot word embeddings @xcite as suggested by @xcite . we use the bi-lstm introduced in the previous section. we refer to the hidden representation of the lstm for readability as h @xmath0, ←h @xmath1. instead of directly applying a fully-connected layer to perform classification based on h i , we first transform h i by an intermediate perceptron unit with relu activation -as shown in 2. the perceptron transforms h i into 20 dimensions, that is, we have @xmath2. our motivation with the extra non-linearity introduced by relu is to encourage an increased diversity in the behavior of the different output units. upon training the lstms, we used the default architectural settings employed by @xcite , i.e., we relied on a word dropout rate of 0.25 @xcite and an additive gaussian noise (with σ = 0.2) over the input embeddings. we trained all our models for 20 epochs using stochastic gradient descent with a batch size of 1. first, we assess the quality of q-mtl towards pos tagging, then we evaluate it on named entity recognition as well. when comparing the performance of different approaches, q-mtl models are compared against the average performance of k stl models, where k denotes the number of task in the case of q-mtl. the k stl models are also used to derive a single prediction by the ensemble model. ## conclusions we proposed quasi-multitask learning (q-mtl), which can be viewed as an efficiently trainable alternative of traditional ensembles. we additionally demonstrated that it acts as an implicit form of regularization as well. in our experiments, q-mtl consistently outperformed the single task learning (stl) baseline for both pos tagging and ner. we have also illustrated that q-mtl generalizes better on smaller and noisy datasets compared to both stl and ensemble models. the computational overhead for the additional classification units in q-mtl is infinitesimal due to the effective aggregation of the losses and the shared recurrent unit between the identical tasks. although we evaluated q-mtl over an lstm, the idea can be applied for more resource-heavy architectures, like transformer @xcite based models where training an ensemble would be too expensive. this is the future direction of our research.