Attention Fusion: a light yet efficient late fusion mechanism for task adaptation in NLU (2022)

Abstract: Fine-tuning a pre-trained language model using annotated data has become the de-facto standard for adapting general-purpose pre-trained models like BERT to downstream tasks. However, given the trend of larger pre-trained models, fine-tuning these models for each downstream task is parameter-inefficient and computationally-expensive deeming this approach sub-optimal for adoption by NLU systems. In recent years, various approaches have been proposed for parameter efficient task adaptation such as Adaptor, Bitfit, Prompt tuning, Prefix tuning etc. However, most of these efforts propose to insert task specific parameters in-between or inside intermediate layers of the pre-trained encoder resulting in higher computational cost due to back-propagation of errors to all layers. To mitigate this issue, we propose a light but efficient, attention based fusion module which computes task-attuned token representations by aggregating intermediate layer representations from a pre-trained network. Our proposed fusion module trains only 0.0009% of total parameters and achieves competitive performance to the standard fine-tuning approach on various tasks. It is also decoupled from the pre-trained network making it efficient during computation and scalable during deployment. Last but not the least, we demonstrate that our proposed attention-fusion mechanism can transfer effectively to different languages for further re-use and expansion.

## introduction aligned with recent advancements in deep learning research, most state-of-the-art (sota) nlu models are built upon neural networks, especially using transformer @xcite based architectures. however, these models require a large amount of domain-specific labeled examples for * equal contribution training, which is prohibitively expensive. the recent adoption of self-supervised pre-training and transfer learning mitigates the issues stemming from scarcity of labeled data @xcite , by pre-training with unsupervised tasks established upon massive unlabeled corpora @xcite @xcite . the resulting models encode syntactic and semantic linguistic information and can be fine-tuned with limited labeled examples on downstream nlu tasks, such as question-answering (qa) @xcite , textual-entailment @xcite , slot labeling (sl) @xcite etc. fine-tuning is a commonly used method that adapts a pre-trained model to a downstream task and has been shown to achieve sota results in various nlu tasks. however, in the presence of larger pre-trained models and many downstream tasks, fine-tuning the whole model for each downstream task is inefficient and expensive due to reasons such as higher memory consumption since gradient and optimizer states need to be stored for all parameters, higher computational cost since error needs to be backpropagated through all layers and bigger cost of hosting large models for each task. parameter efficient domain adaptation has been an area of interest in recent literature comprising of various approaches such as adaptor @xcite , bitfit @xcite , diffprune @xcite , prompt tuning @xcite etc. most of these efforts propose to insert or append task specific parameters in-between or inside of the pre-trained encoder layers, we refer these approaches as early-fusion techniques, as task specific parameters are fused inside the pre-trained network. some drawbacks of early fusion based methods are: during training, loss has to be back-propagated to all layers making them slower; hard to scale in nlu systems as pre-trained encoder and task specific modules are tightly coupled together. in comparison to early fusion, one can place task specific modules after the pre-trained network, so the pre-trained network is untouched regardless of downstream tasks, we refer to this as late-fusion. one late-fusion option is to concatenate @xcite all layers from a pretrained network and project to a lower dimension for task-specific decoders. however, the projection matrices can be considerably big with larger models, e.g., concatenation then projecting hidden layers of a bert-large model to a dimension of 256 amounts to 6.2 million parameters leading to increased computational cost. these challenges hinder the progress of deploying sota transfer learning based models to downstream nlu systems. to address these challenges, we propose attention-fusion, a light but effective task-specific late-fusion based module, for adapting pre-trained models to downstream nlu tasks. our proposed architecture decouples general purpose pre-trained models from downstream task-specific decoder layers with an attention-fusion module. the fusion module enables decoders to effectively adapt hidden representations from intermediate layers of the pre-trained network. to examine the effectiveness of attentionfusion mechanism, we conduct experiments on popular language understanding tasks, including qqp (quora question pair), qnli (questionanswering nli), sst-2 (stanford sentiment treebank), conll-03 (name entity recognition) and a multilingual spoken language understanding (slu) @xcite task using matis dataset. our results demonstrate that attention-fusion module achieves comparable performance to fine-tuning approach while only tuning a small amount of parameters. our attentionfusion approach is a late-fusion based mechanism, thus, exhibiting lower computation cost since backpropagation is limited to task-specific fusion module and decoder layers. furthermore, we empirically show that the task-specific attention-fusion module is transferable across languages. we aim to release our code on github to support further experimentation. in summary, our primary contributions are three-fold: ## related work the importance of efficiently fine-tuning and deploying pre-trained networks to nlu systems has gained wider recognition. in this section, we discuss various approaches proposed in literature. ## approach in this section, we describe the proposed taskspecific fusion model architecture, which augments general-purpose pre-trained models with task-dependent attention on encoded representations in multi-granularity and with prediction layers for nlu tasks. the attention-fusion module aims to improve performance and parameter efficiency by sharing parameters of the pre-trained model with other tasks. ## experiment we evaluate attention-fusion as well as other popular light-weight fine-tuning approaches on 5 popular general language understanding tasks. ## results and discussion in this section, we present the results on monolingual and multilingual tasks, compare the training efficiency and analyze the interactions of attention fusion with different layers across different tasks. ## conclusion with the rising popularity of transfer learning in nlu, the challenge of adapting pre-trained models to nlu tasks effectively and efficiently is becoming increasingly relevant. to address this challenge, we propose a light yet efficient task-specific attention-fusion module which enables parameter sharing and efficient fine-tuning for downstream tasks. we demonstrate that our proposed latefusion module achieves comparable performance to other popular methods as well as the fine-tuning approach, while using less tune-able parameters per task. we also show that the task-specific attentionfusion module is transferable across languages, enabling language expansion work in nlu at a much lower cost.