Hyperdecoders: Instance-specific decoders for multi-task NLP (2022)

Abstract: We investigate input-conditioned hypernetworks for multi-tasking in NLP, generating parameter-efficient adaptations for a decoder using a hypernetwork conditioned on the output of an encoder. This approach produces a unique decoder adaptation for every input instance, allowing the network a larger degree of flexibility than prior work that only produces one decoder adaptation per task. We apply our method to sequence classification tasks, extractive QA, and summarisation and find that it surpasses previous parameter efficient fine-tuning methods and often outperforms fully finetuning the underlying model. An analysis of the embeddings used by our hypernetwork shows that they are sensitive to output label and type, suggesting that our approach better maps from encoder representations to output labels. Our code is publicly available at https://github.com/allenai/hyperdecoders.

## introduction recent work in nlp has examined the performance of large pretrained transformer-based models in multi-task settings, where a single model is evaluated on multiple tasks simultaneously, often with tasks converted to a shared sequence-to-sequence format @xcite . this greatly simplifies model training and deployment, requiring only one deployed model and format to handle multiple tasks. additionally, training across multiple tasks can result in greatly improved performance for similar tasks @xcite , as well as tasks not seen during training @xcite . however, not all tasks work well together, and jointly training on certain task pairs can reduce performance on both ('negative transfer') @xcite . in this paper, we propose a new method for multi-task nlp using a hypernetwork to generate an instance-specific decoder from the output of an encoder. we effectively explore if a model can learn to adapt itself through learning how to generate adapter layers. our approach produces significant gains over prior approaches for efficient multi-task fine-tuning, often matching or exceeding full fine-tuning the underlying model. we build on parameter-efficient learning methods, where one trains a small set of parameters within a much larger model. these parameters may be newly introduced @xcite or already exist within the model @xcite , and are kept as few as possible. this means these methods often do not contain the capacity to learn multiple tasks at once, losing potential transfer benefits. one way to remain parameter-efficient while still handling a variety of tasks is to instead learn to generate these parameters, making use of an auxiliary network ('hypernetwork') to generate the weights used during inference @xcite @xcite . this allows the model to benefit from positive transfer between tasks through the shared hypernetwork while reducing negative transfer by allowing the generated parameters to be unique per task. however, hypernetwork-based approaches generally condition their parameters on a learnt task embedding, meaning (a) the model is the same for every example within a task, and (b) adapting to new tasks requires further training to learn new task embeddings. we increase the flexibility of this approach by instead generating unique parameters for every input, allowing the model to make use of similarities between samples across datasets and avoid potential interference between samples within the same dataset. this is achieved by using a shared encoder across all tasks and then generating adapter layers for the decoder only by feeding the encoded inputs into a hypernetwork. furthermore, by conditioning on inputs rather than task embeddings, our approach allows simple transfer to out-of-domain data, as the shared hypernetwork and encoder learn to map from text to parameters. figure 1 illustrates our approach. we apply our approach to a diverse set of tasks, including sequence classification, extractive question answering, and summarisation, and find that our approach outperforms existing parameterefficient approaches and matches or outperforms full-finetuning. an analysis of our approach shows that sharing parameters in the encoder, but generating them in the decoder is more effective than other possible setups, suggesting that the encoder benefits from multi-task training while the decoder does not. our results suggest that pretrained encoders can be easily adapted and trained to produce adaptations that enhance multi-task transfer learning for decoders, while producing useful adaptations for encoders is much more difficult. to summarise, our core contributions are: ## encoder-conditioned decoders we make use of t5 as the underlying model in our experiments, which is a popular encoderdecoder model for sequence-to-sequence multi-tasking @xcite and a common starting point for previous hypernetwork-based approaches @xcite . however, our overall approach can be applied to generic encoder-decoder transformer models. ## experiments this section details our experimental setup and baselines. results are given in section 4. ## results this section details our main experimental results. ablations and analysis follow in section 5. ## analysis this section presents an analysis of the hyperdecoder's embeddings and ablations, establishing the efficacy of our architecture design choices. ## conclusion we propose a novel method for generating adapters conditioned on a model's input and show that this improves performance in multi-task settings across a variety of tasks. we explore the effectiveness of our approach for sequence classification, qa, and summarisation tasks, and find that it often outperforms strong parameter-efficient baselines. future work could examine applying our approach to other architectures (e.g. decoder-only models) or explore the tradeoffs between shared and generated parameters across different layers. an analysis of our approach suggests the primary benefits come from improved control of the encoder over the decoder, enhancing the effects of positive transfer from the shared encoder. this allows our approach to efficiently adapt a pretrained language model to multiple tasks unseen during pretraining while still benefiting strongly from positive transfer.