Learning to Generate Task-Specific Adapters from Task Description (2021)

Abstract: Pre-trained text-to-text transformers such as BART have achieved impressive performance across a range of NLP tasks. Recent study further shows that they can learn to generalize to novel tasks, by including task descriptions as part of the source sequence and training the model with (source, target) examples. At test time, these fine-tuned models can make inferences on new tasks using the new task descriptions as part of the input. However, this approach has potential limitations, as the model learns to solve individual (source, target) examples (i.e., at the instance level), instead of learning to solve tasks by taking all examples within a task as a whole (i.e., at the task level). To this end, we introduce Hypter, a framework that improves text-to-text transformer’s generalization ability to unseen tasks by training a hypernetwork to generate task-specific, light-weight adapters from task descriptions. Experiments on ZEST dataset and a synthetic SQuAD dataset demonstrate that Hypter improves upon fine-tuning baselines. Notably, when using BART-Large as the main network, Hypter brings 11.3% comparative improvement on ZEST dataset.

## introduction pre-trained text-to-text models @xcite provide a unified formulation and off-the-shelf weights for a variety of nlp tasks, such as question answering @xcite and commonsense reasoning @xcite . in addition to their strong performance, text-totext models naturally support generalizing to novel tasks, by incorporating task description as part of the source sequence and fine-tuning the model with @xcite examples @xcite . at inference time, the model is required to perform 1 code and data can be found at https://github.com/ ink-usc/hypter . ## problem definition we study the problem of learning from task description @xcite , and aim to improve models' competence on unseen tasks at the inference time. formally, a task is denoted as a tuple of (d, d), where d is the natural language description of the task, and @xmath0, t 1 ), ..., (s n , t n )} contains (source, target) examples of this task (see fig. 1 ). in our text-to-text formulation, both s i and t i are text sequences. at train time, both d and d are available, while at test time, an unseen description d is given, and the model is expected to predict the correct t given input s without further training. for instance, in the zest dataset @xcite , a train task description can be "are mountain bikes allowed at this national park?", while d contains twenty paragraphs for different national parks and twenty corresponding answers. during test time, a novel task may be "are there fish in this national park that live in caves?", and the model is asked to directly make inferences. ## background: adapters our work is built on adapters @xcite , light-weight modules that can be placed into transformer layers for parameter-efficient transfer learning. in the original paper, the main model is frozen during training, while only layer norm and adapter parameters are learnable. in this paper, we adopt a simplified design compared to the original paper (see fig. 2 (left)) -in each transformer layer, exactly one adapter module will be added after the multi-headed attention. one adapter module contains two linear layers separated by an nonlinearity activation layer. we use (w id , b id ) to denote the down-projection parameters for the adapter in transformer layer i, and (w iu , b iu ) for the upprojection parameters. ## method overview. fig. 2 provides an illustration of our hypter framework. hypter has two major parts: (1) a main network, which is a pre-trained text-totext model. we instantiate the main network with bart-base/large @xcite . (2) a hyper-network, which generates adapters to be plugged into the main network. fig. 2 (left) contains a detailed illustration of how adapter parameters are generated and how adapter layers are incorporated into one transformer layer. hypernetwork. the hypernetwork consists of an encoder and multiple decoders. the encoder maps the task description d to a latent representation h 0 , while the decoders use h 0 to generate adapter parameters φ. in our work we instantiated the encoder with a roberta-base model @xcite , i.e., h 0 = roberta(d). for a textto-text model with n transformer layers, the hypernetwork contains n decoders. decoder i uses h 0 as input, and outputs adapter parameters φ i for trans- ## conclusion in this paper, we introduced hypter, a framework to improve text-to-text transformer's generalization ability to unseen tasks. hypter enhances taskspecific abilities by inserting adapters generated with a hypernetwork, meanwhile it maintains the model's general task-solving ability by freezing main model parameters. we demonstrated the effectiveness of hypter on two datasets. future work may explore teaching models with compositional instructions using hypter, or propose robust fine-tuning methods that help the model generalize to unseen data. it is also necessary to construct a large dataset of diverse nlp tasks to facilitate future research in this direction. a dataset details zest. zest dataset is released at https: //ai2-datasets.s3-us-west-2.amazonaws.com/zest/ zest.zip. zest leaderboard is hosted at https: //leaderboard.allenai.org/zest/submissions/public .