Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020)

Abstract: Fine-tuning pre-trained generative language models to down-stream language generation tasks has shown promising results. However, this comes with the cost of having a single, large model for each task, which is not ideal in low-memory/power scenarios (e.g., mobile). In this paper, we propose an effective way to fine-tune multiple down-stream generation tasks simultaneously using a single, large pretrained model. The experiments on five diverse language generation tasks show that by just using an additional 2-3% parameters for each task, our model can maintain or even improve the performance of fine-tuning the whole model.

## introduction large-scale language models @xcite have shown to be effective in learning highly transferable embedding, which can be used in several down-stream tasks. for instance, bidirectional models @xcite are fine-tuned to improve classification tasks @xcite , while, unidirectional language models @xcite are more effective in language generation tasks. in this work, we focus on the latter, and show that it is possible to dynamically steer the output of a language model (e.g., gpt-2) towards a specific task (e.g., summarization) without modifying the original model parameters. feature-based transfer @xcite and fine-tuning @xcite are the most commonly used methods for transfer learning of a language. the former freezes the pre-trained model and uses it as a feature extractor for training a new classifier, and the latter uses the pre-trained weight as a weight initialization for the model to be trained for downstream tasks. the feature-based transfer strategy has not shown promising results @xcite , while fine-tuning, on the other hand, can achieve state of the art performance in multiple tasks @xcite . however, the downside of the latter is the need for a seperate model for each of the fine-tuned tasks. this is especially relevant for on-device applications, where a limited amount of computation/memory is available. therefore, we study how to effectively use a single pre-trained model as the backbone for multiple language generation tasks, such as conversational question answering, summarization, machine translation, multi-turn chit-chat dialogue, and task-oriented natural language generation. this is a particular parameter-sharing schema, where we constrain the shared parameters to be the ones in the pre-trained model, and we learn task-specific parameters for each of the considered datasets. in this paper, we propose to use residual adapter layers @xcite and task embeddings for modelling the aforementioned task-specific parameters, and we explore different training strategies such as distillation @xcite . we also analyse the trade-off between freezing or not freezing the language model parameters by leveraging two learning settings, multi-task (mt) @xcite and continual learning (cl) @xcite . with our experiments, we empirically demonstrate that by adding less than 3% task-specific parameters, our model can maintain or even achieve better performance than fine-tuning the whole model. ## related work pre-trained generative language models @xcite @xcite @xcite have shown to be very effective in language generation, whereas, bidirectional pre-trained models @xcite @xcite significantly improve the performance of several down-stream classification tasks. fine-tuning large pre-trained models has shown positive results in dialogue tasks @xcite and other language generation tasks @xcite . however, all of the previous works only consider fine-tuning on each generation task individually, which requires a separate model for each task. in this work, we use only a single model, for multiple generation tasks. residual adapters, derived from residual networks @xcite , were first introduced by @xcite for multiple visual domain learning. @xcite proposed low-rank residual adapters to improve the scalability of the adapter module, and effectively transfer bert @xcite to multiple text classification tasks simultaneously, while @xcite applied an adapter layer to language/domain adaptation for neural machine translation. on the other hand, @xcite proposed a plug and play method to control the language model generation without finetuning the model. differently, in this paper, we extend the idea of adapters to a large variety of language generation tasks, which has not been considered before, and we compare the idea of a fixed pre-trained back-bone for continual learning with multi-task training (stickland and @xcite . ## methodology the versatile language model (vlm) is composed of three components: a pre-trained language model back-bone (e.g., gpt-2), and two kinds of specialized parameters for each generation tasks such as low-rank residual adapters and task embedding. figure 1 shows the vlm architecture with the specialized parameters in different colours. residual adapters these are trainable modules which steer the pre-trained model to different downstream tasks. we adapt the design of the feedforward transformer sub-layer following @xcite . to elaborate, the adapter block consists of 1) a layer normalization @xcite for an efficient adaptation and 2) a following autoencoder @xcite , with a residual connection. formally, given the hidden representation h @xmath0, where d is the hidden dimension and t is the current generation step, the residual adapter computes the following: task embedding. to adapt unconditional generative language models to different conditional language generation tasks (e.g., coqa, summarization), we construct a set of task-specific segment embeddings. for example, in multi-turn dialogue, we alternate between system and user embeddings to help the model to capture the hierarchical structure of dialogues. figure 1 shows the task embedding for each task, and more details are available in appendix a2. ## conclusion in this paper, we have presented a versatile language model which learns five diverse natural language generation tasks in a single model. we found that a residual adapter is more effective than finetuning other parts of the model (e.g., lm-head), and that distillation helps in reducing the gap in performance in hard to fine-tune tasks, such as summarization and translation. finally, we show the trade-off between a frozen and trainable back-bone, showing that the former has a competitive performance, with the advantage of being extendable to future tasks without full re-training. ramakrishna vedantam, c lawrence zitnick, and devi parikh. 2015. cider: consensus-based image description evaluation. in proceedings of the ieee conference on computer vision and pattern recognition, pages 4566-4575. alex wang, amanpreet singh, julian michael, felix hill, omer levy, and samuel r. bowman. 2019. glue: a multi-task benchmark and analysis platform for natural language understanding. in international conference on learning representations. yijun wang, yingce xia, li zhao, jiang bian, tao qin, guiquan liu, and tie-yan liu. 2018. dual transfer learning for neural machine translation with marginal distribution regularization. in thirty-second aaai conference on artificial intelligence. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, r'emi louf, morgan funtowicz, and jamie brew. 2019a. huggingface's transformers: state-of-the-art natural language processing. arxiv, abs/1910.03771. thomas wolf, victor sanh, julien chaumond, and clement delangue. 2019b. transfertransfo: a transfer learning approach for neural network based conversational agents. corr, abs/1901.08149. thomas wolf, victor sanh, julien chaumond, and clement delangue. 2019c. transfertransfo: a transfer learning approach for neural network based conversational agents. arxiv preprint arxiv:1901.08149. felix wu, angela fan, alexei baevski, yann n dauphin, and michael auli. 2019. pay less attention with lightweight and dynamic convolutions. arxiv preprint arxiv:1901.10430. zhilin yang, zihang dai, yiming yang, jaime carbonell, ruslan salakhutdinov, and quoc v le. 2019. xlnet: generalized autoregressive pretraining for language understanding. arxiv preprint arxiv:1906.08237. semih yavuz, abhinav rastogi, guan-lin chao, and dilek hakkani-tur. 2019. deepcopy: grounded response generation with hierarchical pointer networks. arxiv preprint arxiv:1908.10731. saizheng zhang, emily dinan, jack urbanek, arthur szlam, douwe kiela, and jason weston. 2018. personalizing dialogue agents: i have a dog, do you have pets too? in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2204-2213. chunting zhou, graham neubig, and jiatao gu. 2019. understanding knowledge distillation in nonautoregressive machine translation. arxiv preprint arxiv:1911.02727.