Maximal Multiverse Learning for Promoting Cross-Task Generalization of Fine-Tuned Language Models (2021)

Abstract: Language modeling with BERT consists of two phases of (i) unsupervised pre-training on unlabeled text, and (ii) fine-tuning for a specific supervised task. We present a method that leverages the second phase to its fullest, by applying an extensive number of parallel classifier heads, which are enforced to be orthogonal, while adaptively eliminating the weaker heads during training. We conduct an extensive inter- and intra-dataset evaluation, showing that our method improves the generalization ability of BERT, sometimes leading to a +9% gain in accuracy. These results highlight the importance of a proper fine-tuning procedure, especially for relatively smaller-sized datasets. Our code is attached as supplementary.

## introduction recently, there has been an increasing number of studies suggesting the use of general language models, for improving natural language processing tasks @xcite @xcite . among the most promising techniques, the unsupervised pretraining approach (dai and le, 2015; @xcite has emerged as a very successful method, that achieves state-of-the-art results on many language tasks, including sentiment analysis @xcite , natural language inference @xcite and similarity and paraphrasing tasks @xcite . this approach incorporates a two-phase training procedure. the first phase utilizes an unsupervised training of a general language model on a large corpus. the second phase applies supervision to fine-tune the model for a given task. more recently, unsupervised pretraining models such as bert @xcite , xlnet @xcite and roberta @xcite , have achieved unprecedented performance. for example, in the glue benchmark @xcite , bert @xcite was reported to achieve performance that exceeds human level on a few different datasets, such as qnli @xcite , qqp @xcite and @xcite . however, despite the great progress achieved by these taskspecific and dataset-specific models, it is not yet clear how well they can generalize to different tasks, how well they generalize when evaluating the same task on different datasets, and how to improve this generalization ability. in our work, we extend the multiverse method of @xcite , which was shown to improve transfer learning in the computer vision task of face recognition and on the cifar-100 small image recognition dataset. the multiverse loss generalizes the cross entropy loss, by simultaneously training multiple linear classification heads to perform the same task. in order to prevent multiple copies of the same classifier, in the multiverse scheme, each classifier is mutually orthogonal to the rest of classifiers. the number of multiverse heads used was limited, never more than seven and typically set to five. we propose a novel fine-tuning procedure for enhancing the generalization ability of the recent unsupervised pretrained language models, by employing a large number of multiverse heads. the essence of our technique is as follows: given a pretrained language model and a downstream task with labeled data, we fine-tune the model using a maximal number of multiverse classifiers. the fine-tuning goal is to both minimize the task loss and an orthogonality loss applied to the classification heads. when enforcing orthogonality hinders the classifiers' performance, we detect and eliminate the less effective classification heads. the technique, therefore, preserves a maximal set of classifiers, which is comprised of the best performing ones. by maintaining this maximal subset during training, our method leverages multiverse loss to its fullest. hence, we name our method maximal multiverse learning (mml). our contributions are as follows: (1) we present mml, a general training procedure to improve the transferability of neural models. (2) we apply mml on bert and report its performance on various datasets. (3) we propose a set of cross dataset evaluations using common nlp benchmarks, demonstrating the effectiveness of mml, in comparison to regular bert fine-tuning and to alternative regularization techniques. ## related work many recent breakthroughs in nlp employ unsupervised pretraining of language models. the different variants can be categorized into two main approaches: (1) feature-based models, such as @xcite and (2) fine-tuning models, such as @xcite @xcite . the former technique utilizes a language neural-based model as a feature extractor. the extracted features may be used for the training of another model, receiving the extracted features as input. the second approach utilizes a similar pre-trained model, but fine-tunes it in an end-to-end manner to specialize on a given task. during the fine-tuning phase, all of the parameters of the model are updated and a relatively small number of parameters are trained from scratch. the usage of multiple classifiers can be found in few places in the literature. in googlenet @xcite , the authors use multiple classifier heads in different places in the model architecture. the additional classifiers led to better propagation of the gradients during training. however, with the advent of better conditioning and normalization methods, as well as with the modern introduction of skip connections in architectures such as the resnet @xcite , the practice of adding intermediate branches, for the sake of introducing an auxiliary loss at lower layers, was mostly abandoned. the multiverse loss was shown to promote transfer learning and to lead to a low-dimensional representation in the penultimate layer @xcite . however, the current literature does not present any methodological way to select the number of multiverse heads and the idea was only applied for a handful of parallel classifiers. in mml, hundreds of multiverse heads are used, leading to a tradeoff between the classifier accuracy and the orthogonality constraint. mml balances the two terms by pruning, during training, the under-performing heads. ## method let @xmath0. let y be the set of all possible sentences generated by w , including the empty sentence. a language model m : y × @xmath1. given a dataset with n training samples, s 1 ...s @xmath2, each associated with a label y @xmath3...c], we denote the coding vector of each sample by d i := m (s i ). as a concrete example, for the bert model, d i is the latent embedding of the cls token. common language models employ a classifier c : r @xmath4, the output of c is a logit vector, and pseudoprobabilities are obtained by applying softmax p i y @xmath5. different from the single-classifier models, our model utilizes a multiverse classifier c : r @xmath6, ...,c m (d i )), where m is a multiplicity parameter, c j : r @xmath7, each with different weights, applying the same function as eq. 1. : as a set of binary scalars. each classifier head c j will be associated with a different binary scalar β j , which is set during training. in our experiments, we set m to be equal to the coding vector size d , which entails a full rank of active multiverse classifiers at the beginning of the training. the loss function is composed of two components, the task loss and the multiverse loss. defining active multiverse classifiers as those that are associated with a value β @xmath8, the task loss optimizes the performance of all active multiverse the task loss comprises a loss-term for each multiverse classifier, using the given labels of the task at hand. the mutual orthogonality tables hold the absolute value of the dot product calculated between the weights of all classifiers, across the different classes. since orthogonality of a classifier with itself is ignored, we set the diagonal to 0. following multiverse loss definition and since orthogonality is symmetric, only half of each table value is passed to the multiverse loss. algorithm 1 mml training. the meanshift function returns the center of the clusters. params: @xmath9, γ = 0.99, t hr eshol @xmath10, α = 2 • e -5 , λ = 0.005. ∀1 ≤ @xmath11: β j ← 1, a j ← 0 for st @xmath12, 2, ... do sample a minibatch {(s i , as mentioned earlier, each classifier c j is associated with a binary value β j , which controls the applicability of the classifier and is configured during training. under the context of the loss function, setting β j to 0 would eliminate the impact of the j t h classifier head c j for both the task loss and multiverse loss. for a multi-class classification task, we apply the following task loss: the second loss term enforces orthogonality between the set of classifiers, for each class separately, using the multiverse loss: the total loss l t ot al is defined as: l t ot @xmath13. similar to @xcite , we set λ = 0.005, throughout all of our experiments. the mml model is illustrated in fig. 1 . the training algorithm begins with an initialization of the aggregated model (m , c , b ). m may be initialized by any pre-trained general language model. the multiverse classifiers are randomly initialized from scratch, and all classifiers are initially activated, by setting β @xmath14. during training, we track the performance of each multiverse classifier separately. every k steps, we search for a subset of the topperforming classifiers. when we find such a subset, we eliminate the less performing classifiers by setting their corresponding βs to 0. in order to detect the top-performing classifiers, we calculate a moving average variable a j for each multiverse classifier. specifically, a j holds the moving average of the task loss value l j t ask associated with classification head c j . a j is being updated for every training step, using the moving average momentum constant of 0.99. every k steps, we run the meanshift algorithm @xcite on the set a j |β @xmath15. meanshift is a clustering algorithm that reveals the number of clusters in a given data, and retrieves the corresponding centroid for each detected cluster. in our case, mean-shift is applied on 1d data and by utilizing it, we identify the subset of top-performing multiverse heads as the cluster associated with the minimal centroid value. next, we eliminate the rest of the multiverse heads, by setting their corresponding β to 0. this adaptive elimination is stopped, when we reach a minimal number of active heads, see alg. 1. at inference, we use the active multiverse heads to retrieve predictions. specifically, given a sample s i , we calculate the logits ŷ as: for classification tasks, we apply the softmax function on ŷ, and return its output. for regression tasks, we simply return ŷ. ## results in this study, we evaluate mml, applied on a pretrained bert @xcite ) model, using nine nlp datasets, while employing two different settings: (1) fine-tuning on different downstream tasks from the glue benchmark @xcite , and (2) cross dataset evaluations for different datasets of the same or similar tasks. for the first, we fine-tune mml on each dataset separately, and evaluate its performance on the development set and the test set of the same dataset. for the second, we evaluate our fine-tuned mml models on the train and development set of other datasets within the same task category. this allows us to study the generalization level of all models, across different datasets. additionally, we perform an ablation study and report empirical results that showcase the efficacy of mml, compared to other multiverse schemes and to a bert model with a higher dropout rate. we adopt eight datasets from the glue benchmark @xcite and one extra dataset supporting the task of natural language inference (nli). the datasets can be arranged to form the following three categories: (1) inference tasks: this category incorporates four datasets of natural language inference: rte, mnli, snli and qnli. rte and qnli are binary classification tasks, whereas mnli and snli are multilabel classification tasks (which possess an additional "neutral" label). (2) similarity and paraphrasing tasks: this category includes three datasets: mrpc, qqp, sts-b. mrpc and qqp are binary classifications tasks, while sts-b is a regression task with labels annotating the level of similarity between each sentence pair. (3) misc. datasets: this category composed of two datasets that cannot be used for the cross dataset evaluation, due to the lack of commonality between their tasks: cola and sst-2. we refer the reader to the appendix for more details on the nine datasets. ## discussion the results in both tab. 2 and 3, reveal a significant gap in performance for all models when evaluated on holdout datasets, although the holdout datasets share the same or similar task each model was trained for. for example, both mml-mrpc and bert-mrpc models yield a ∼20% degradation in absolute accuracy on the qqp dataset. however, compared to bert, our mml method produces significantly better performance on the cross evaluations. specifically, when evaluated on qqp, mml-mrpc outperforms bert-mrpc by a relative improvement of ∼4.6%, for both development and train set. we do not observe a direct link between the improvement obtained on the same dataset evaluation to that obtained in the cross dataset one. for example, our mml-qnli model was able to outperform bert-qnli in the cross dataset evaluation, although, compared to our bert-qnli reproduction, mml-qnli exhibits a somewhat degraded performance on qnli's development set (see the third section in tab.2) and test set (as published in @xcite ). we attribute this to the ability of mml to encourage the model to produce more transferable coding vectors. computational overhead during training, the mml multiverse heads imply a maximal addition of 1024 operations of matrix multiplications and gradient derivations, each in the size of 1024 × c. during the backward steps, the multiple gradients are averaged almost immediately, right before propagating them back through the last encoder layer of bert. interestingly, in many cases, the number of active heads shrinks in the early stages of the training. during inference, the additional calculations are solely the matrix multiplications, for which the average number of active heads, across our experiments, is 381. this average number of active heads translates to an increase of ∼0.2% to the parameter count of bert_large. ## summary we introduce the mml method for fine-tuning language models. mml utilizes a large set of parallel multiverse heads and eliminates the relatively weaker heads during training. we demonstrate the effectiveness of mml on nine common nlp datasets, by applying inter-and intradatasets evaluation, where it is shown to outperform the originally introduced bert fine-tuning procedure. the results shade light on the role of regularization in improving cross task generalization, and show the advantage of mml over alternative regularization methods. erhan, vincent vanhoucke, and andrew rabinovich. 2015. going deeper with convolutions. in