Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021)

Abstract: State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. However, such modules are trained separately for each task and thus do not enable sharing information across tasks. In this paper, we show that we can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model. This parameter-efficient multi-task learning framework allows us to achieve the best of both worlds by sharing knowledge across tasks via hypernetworks while enabling the model to adapt to each individual task through task-specific adapters. Experiments on the well-known GLUE benchmark show improved performance in multi-task learning while adding only 0.29% parameters per task. We additionally demonstrate substantial performance improvements in few-shot domain generalization across a variety of tasks. Our code is publicly available in https://github.com/rabeehk/hyperformer .

## introduction transfer learning from pretrained large-scale language models yields state-of-the-art results in a variety of tasks @xcite @xcite . as a highly expressive and abstract framework, @xcite explored the landscape of transfer learning by converting text-based natural language processing (nlp) problems into a sequence-to-sequence format to train a unified model on several tasks simultaneously. multi-task learning with pretrained language models @xcite is appealing for multiple reasons: 1) training individual models per task results in higher computational costs, which hinders deployment and maintenance. these costs are substantially reduced by training a single * work done while the author was at google. our hyperformer adapter architecture. following @xcite , we include adapter modules after the two feed-forward layers. the adapter hypernetwork h l a produces the weights (u l τ and d l τ ) for task-specific adapter modules conditioned on an input task embedding i τ . similarly, the layer normalization hypernetwork h l ln generates the conditional layer normalization parameters (β τ and γ τ ). during training, we only update layer normalizations in t5, hypernetworks, and task embeddings. the compact hyperformer++ shares the same hypernetworks across all layers and tasks and computes the task embedding based on task, layer id, and position of the adapter module ( §2.4). model. 2) fine-tuning the model across multiple tasks allows sharing information between the different tasks and positive transfer to other related tasks. specifically, when target datasets have limited training data, multi-task learning improves the performance compared to individually trained models @xcite . however, multi-task fine-tuning can result in models underperforming on high-resource tasks due to constrained capacity @xcite . an additional issue with multi-task fine-tuning is the potential for task interference or negative transfer, where achieving good performance on one task can hinder performance on another @xcite . ## hyperformer in this section, we present our hyperformer model, which integrates hypernetwork-based adapter layers into a multi-task transformer model. in §2.4, we introduce a parameter-efficient variant of this model, called hyperformer++. ## conclusion we propose a parameter-efficient method for multi-task fine-tuning. our approach is to train shared hypernetworks to generate task-specific adapters conditioned on the task, layer id, and adapter position embeddings. the shared hypernetworks capture the knowledge across tasks and enable positive transfer to low-resource and related tasks, while task-specific layers allow the model to adapt to each individual task. extensive experiments show that our method obtains strong improvement over multi-task learning on the glue benchmark, and substantially improves the in-domain task generalization. for mnli, we report accuracy on the matched validation set. for mrpc and qqp, we report accuracy and f1. for sts-b, we report pearson and spearman correlation coefficients. for cola, we report matthews correlation. for all other tasks, we report accuracy.