Boosting Natural Language Generation from Instructions with Meta-Learning (2022)

Abstract: Recent work has shown that language models (LMs) trained with multi-task instructional learning (MTIL) can solve diverse NLP tasks in zero- and few-shot settings with improved performance compared to prompt tuning. MTIL illustrates that LMs can extract and use information about the task from instructions beyond the surface patterns of the inputs and outputs. This suggests that meta-learning may further enhance the utilization of instructions for effective task transfer. In this paper we investigate whether meta-learning applied to MTIL can further improve generalization to unseen tasks in a zero-shot setting. Specifically, we propose to adapt meta-learning to MTIL in three directions: 1) Model Agnostic Meta Learning (MAML), 2) Hyper-Network (HNet) based adaptation to generate task specific parameters conditioned on instructions, and 3) an approach combining HNet and MAML. Through extensive experiments on the large scale Natural Instructions V2 dataset, we show that our proposed approaches significantly improve over strong baselines in zero-shot settings. In particular, meta-learning improves the effectiveness of instructions and is most impactful when the test tasks are strictly zero-shot (i.e. no similar tasks in the training set) and are “hard” for LMs, illustrating the potential of meta-learning for MTIL for out-of-distribution tasks.

## introduction given some basic instructions and a few demonstrations, humans are capable of conducting diverse tasks without any supervision. can language models perform similarly on unseen tasks when trained with instructions? specifically can such an approach work on complex generation tasks with relatively smaller language models (lms)? the recent advances in large lms have shown tremendous potential in diverse ai applications and have the capability to change the way model developers and users interact with intelligent systems. the inherent representative power of such models has shown that diverse nlp tasks can be solved purely by appending prompts or demonstrations in context before a test input @xcite . this has led to the rise of prompt-based training @xcite where even much smaller models trained on a large set of tasks in a multi-task setting with prompts, can behave similarly @xcite . a natural extension of the prompt tuning concept involves adding instructions about the task along with the demonstrations. instructions are more informative than prompts and aid the language models to solve unseen tasks better. instructions can have different forms, for example to convey a short task specific statement (e.g. "provide a short summary for the following input") @xcite , or a natural language question ("how would you rephrase that in a few words?") @xcite @xcite . however, for complex generation tasks, short instructions can be ambiguous and non-informative and thus need large lms which can encode a much richer prior knowledge. in contrast, @xcite define instructions in the natural instructions v2 (niv2) dataset comprising of detailed task descriptions, positive and negative examples, and explanations. instructions in niv2 are similar to annotation guidelines, and thus potentially more beneficial foot_0 . using multi-task instructional learning (mtil) on diverse tasks, @xcite showed that even smaller models can be competitive with larger models on zero-shot generalization to unseen tasks. results in @xcite illustrated that lms can extract useful information from instructions beyond the surface patterns available in the prompts for solving a task. this suggests that learning-to-learn or meta-learning paradigm can further enhance the utilization of instructions by learning about task at deeper levels. in this paper, we investigate how smaller lms could best benefit from the natural instructions and whether meta-learning paradigms can further improve the zero-shot generalization ability of lms in mtil. meta-learning has been shown to be effective in adapting knowledge with little supervision but to the best of our knowledge has not been adapted to mtil in zero-shot settings. specifically, we explore two different metalearning approaches. first we propose to adapt model agnostic meta learning (maml) @xcite for mtil, an optimization based approach. second, we explore hyper-network (hnet) @xcite based mtil, a black-box approach. hnet introduces an auxiliary lm which encodes instructions to produce task specific parameters which are added to the main lm parameters to generate a task specific lm at prediction time. in addition, we evaluate a third approach which combines the two into a hnet-maml by training the hnet model using maml. we conduct extensive experiments specifically designed to test the generalization ability of lms trained with instructions under different zero shot conditions. we use two sets of training tasks from the niv2 dataset: 1) all natural language tasks and 2) natural language generation tasks. we evaluate the models for two sets of held out generation tasks conveying different levels of zero-shot generalization ability: 1) weak generalization set with a random selection of generation tasks with potential overlap of categories with training tasks and 2) strong generalization set (or strict zero-shot conditions) using summarization and title generation tasks with no overlap in categories from the training tasks. we further investigate the task sets under difficulty levels of easy, medium, and hard based on their baseline rouge scores. the main conclusion from our study is that under strict zero-shot conditions, meta-learning with instructions significantly improves the performance. the improvements become more significant for the strong generalization task set and when the task difficulty level is hard (i.e. tasks where the lm struggles to generate correct outputs in zero-shot setting). moreover, meta-learning increases the effectiveness of instructions under all conditions. while both maml and hnet models show im-provements over the baselines, hnet (along with its maml extension) by explicitly enforcing the use of instructions through task specific conditioning of parameters, results in larger gains. in summary, the main contributions of the paper are two-fold. first, we adapt meta-learning approaches to mtil. second, we study their efficacy and show significant improvements under strict zero-shot conditions. ## related work learning from instructions: an extension of the basic prompt-based in-context learning is appending task specific instructions with prompts. several recent works which include flan @xcite , t0 @xcite and @xcite , train a large lm in a multi-task setting with instructions. instructgpt @xcite takes slightly different approach by training the gpt3 model @xcite with human annotated dataset of demonstrations of desired user intents and use reinforcement learning to improve the model to follow such instructions. yet another direction called pattern-exploiting training (pet) @xcite combines the idea of formulating instructions as cloze questions and show that even small lms can be good few-shot learners and work with language generation. meta-learning for language generation: meta learning has been applied in several language generation settings such as @xcite to induce persona in a chatbot, @xcite for task oriented dialog systems, @xcite for low resource machine translation, and @xcite for abstractive summarization in a low-resource transfer learning but do not use instructions for zero-shot transfer. our mtil scenario is closely related to metaicl @xcite which applies multi-task learning in-context in a k-shot setting for classification tasks, but differs in that it is a k-shot in-context scenario and does not use instructions or meta-learning optimization. while these works are related, to the best of our knowledge, meta-learning has not been used to generalize to unseen generation tasks in zero shot settings using instructions and thus the paper provides several novel insights and approaches. hyper-networks (hnet) in nlp applications: @xcite use hnet to train lms in a multi-task setting with adapters and @xcite propose a contin-ual learning framework with hnets conditioned on unique task ids to reduce catastrophic forgetting. hnets have been used for input conditioning a decoder in @xcite which produces a unique decoder for each input, and thus is similar to our approach. however these the approaches are not strictly applicable in our zero-shot scenario or in general nlp tasks with task descriptions in natural language. language model editing: our hnet based approach is based on the architecture in @xcite which uses it to edit factual knowledge in lms. while the architecture is similar, we use the hnet to encode task specific instructions and is intended for controlling task-level lm behavior unlike the micro-behavior targeted in @xcite . similar to ours and @xcite , bayesian hyper networks @xcite linearizes the number of parameters for predictions by constraining the hnet outputs to scale and shift parameters. @xcite propose meta learning approaches for editing errors in a neural network but is not directly applicable for mtil in a zero-shot setting. mtil: finally, the work most closely related to this paper is the tk-instruct model from @xcite which fine tunes a t5 model @xcite with instructions, which we use as the baseline. we use the same dataset and training settings as tk-instruct but instead use the pretrained bart model @xcite as it is task agnostic compared to t5 (t5 may not represent a true zero-shot setting). in addition, we enhance this model with meta-learning and consider significantly different training, evaluation, and model settings to test zero-shot generalization resulting in unique contributions and conclusions orthogonal to the findings in @xcite . ## problem setup in this section we briefly outline our problem settings and baselines used in this paper. ## meta-learning with instructions training on a large number of diverse tasks and testing on unseen tasks lend itself to the paradigm of learning-to-learn or meta-learning, which has been successfully applied for generalizing in both zero-and few-shot scenarios. task meta-data in the form of instructions can also provide discriminative information about the task process in addition to the surface patterns of the input and output strings. we investigate whether meta-learning can aid such learning and adapt three approaches to mtil. ## conclusions in this paper we investigate whether meta-learning applied to multi-task instructional learning (mtil) can boost the generalizability of lms to unseen tasks in a zero-shot setting. specifically, we eval- figure 6: % differences from standard training with other models with bart-large. hnet has the best performance for the strong generalization set. while the models perform relatively well under zero-shot conditions, the performance is far from fully supervised models. it remains to be seen at what point we can match fully supervised models (for example using a k-shot setting). in addition, the impact of hnet-maml on the bart-large model was lower. it will be interesting to see how meta-learning scales with model sizes and whether the additional bandwidth from larger models can negate the impact of meta-learning in encoding and utilizing instructions. this is subject of future work.