Low-Rank Updates of pre-trained Weights for Multi-Task Learning (2023)

Abstract: Multi-Task Learning used with pre-trained models has been quite popular in the field of Natural Language Processing in recent years. This framework remains still challenging due to the complexity of the tasks and the challenges associated with fine-tuning large pre-trained models. In this paper, we propose a new approach for Multi-task learning which is based on stacking the weights of Neural Networks as a tensor. We show that low-rank updates in the canonical polyadic tensor decomposition of this tensor of weights lead to a simple, yet efficient algorithm, which without loss of performance allows to reduce considerably the model parameters. We investigate the interactions between tasks inside the model as well as the inclusion of sparsity to find the best tensor rank and to increase the compression rate. Our strategy is consistent with recent efforts that attempt to use constraints to fine-tune some model components. More precisely, we achieve equivalent performance as the state-of-the-art on the General Language Understanding Evaluation benchmark by training only 0.3 of the parameters per task while not modifying the baseline weights.

## introduction multi-task learning (mtl) aims in exploiting simultaneously similarities and differences between related tasks @xcite . compared to training the models separately, this can lead to enhance learning efficiency and prediction accuracy for the task-specific models. in addition to certain similarities to transfer learning and data augmentation, mtl has a regularizing effect in practice @xcite . mtl also has the advantage of storage efficiency, which is advantageous for devices with less memory. on the other hand, mtl performance may be impacted by task covariance @xcite , various loss functions, and difference between dataset sizes @xcite . additionally, there are still some mtl-related limitations, including negative transfer, in which learning two tasks at once lowers the model's performance on both tasks @xcite , and catastrophic forgetting in which one some tasks features can be overlooked during the training process @xcite . in this study, we aim to decrease the amount of language model trainable parameters in the mtl framework. to achieve this, we suggest stacking weight matrices corresponding to several tasks in a 3-way tensor and performing a tensor low-rank update, which is similar to the lora technique in the single-task case @xcite . one of the main advantages of the tensor approach is that it allows for splitting the weight updates into shared and task-specific parts. moreover we extend our approach to the bias term which showed remarkable results in ben @xcite . we test our method using the general language understanding evaluation (glue) benchmark. thus we demonstrate that low-rank update for both matrix and bias successfully strikes a balance between preserving positive transfer and minimizing negative transfer by only training 0.3% of the initial parameters per task. we also look into how different model factors affect the way tasks interact with one another. ## related work multi-task learning for nlp. training the bidirectional encoder representations from transformers (bert) @xcite in hard-sharing multi-task learning may be subject to negative transfer @xcite . to tackle this problem, some studies propose to use a shared hyper-network or to do conditional learning @xcite @xcite . this method consists in creating a task embedding which is used to build model's layers. another approach to circumvent the negative transfer problem, consists to use knowledge distillation (kd) where single-task teachers transfer their knowledge to one multi-task student @xcite . tensor methods. the use of tensor methods mainly focused on applying tensor approximations for compression of pretrained models (compression of fully-connected @xcite and convolutional networks @xcite ; @xcite ). @xcite utilized tensor decomposition for compressing pre-trained language models and presented a formal framework with defined nomenclature to thoroughly explore tensor decomposition approaches to compress transformerbased language models. in multi-task learning, tensor methods have been used to introduce sharing between weights across different tasks @xcite @xcite . recent work considered splitting in task-agnostic (shared) and taskspecific parts. however, the cited works were mostly focused on learning compressed representation or tensor completion, mostly with so-called tucker tensor decomposition. the weight representation in our work is much simpler and more efficient in terms of parameters: the same frozen weight matrix is shared and task-specific updates use canonical polyadic decomposition (cpd). it is compact and have an additional interpretation with shared and task-specific factors. we can even do it with tucker (see more details on cpd in appendix a.1. ## morris: multi-task learning based on low-rank updates of pre-trained weights in the following, we designate vectors, matrices, and tensors, respectively, with bold lowercase letters, bold capital letters, and calligraphy letters. we assume that there are t tasks and t associated datasets ## experiments and analyse we shall now present our experimental results. ## conclusion in this paper we presented, a novel method for multi-task learning that relies on stacking the neural network weights into a tensor. we demonstrated that low-rank updates in the conventional polyadic tensor decomposition of this tensor of weights result in an efficient technique that allows for a significant reduction in model parameters without sacrificing performance. on the glue benchmark, we showed that our proposed approach successfully achieves a compromise between maintaining positive transfer and reducing negative transfer by only using 0.3% of the initial model's parameters.