Multi-Task Transfer Matters During Instruction-Tuning (2024)

Abstract: Instruction-tuning trains a language model on hundreds of tasks jointly to improve a model’s ability to learn in-context;however, the mechanisms that drive in-context learning are poorly understood and, as a result, the role of instruction-tuning on in-context generalization is poorly understood as well.In this work, we study the impact of instruction-tuning on multi-task transfer: how well a model’s parameters adapt to an unseen task via fine-tuning.We find that instruction-tuning negatively impacts a model’s transfer to unseen tasks, and that model transfer and in-context generalization are highly correlated, suggesting that this catastrophic forgetting may impact in-context learning.We study methods to improve model transfer, finding that multi-task training—how well the training tasks are optimized—can significantly impact ICL generalization; additionally, we find that continual training on unsupervised pre-training data can mitigate forgetting and improve ICL generalization as well.Finally, we demonstrate that, early into training, the impact of instruction-tuning on model transfer to tasks impacts in-context generalization on that task.Overall, we provide significant evidence that multi-task transfer is deeply connected to a model’s ability to learn a task in-context.

## introduction the surprising ability of large language models to follow natural language instructions and perform a variety of complex tasks is attributed in large part to instructing tuning, the process of fine-tuning a language model with supervised demonstrations of instructions or in-context examples paired with the desired output @xcite . however, the mechanisms behind the success of instruction tuning remain poorly understood. in principle, a good in-context learner may perform a task by implementing a learning algorithm over the examples passed in-context @xcite ; in practice, it is not clear how much of the task is learnt versus recognized via memorization or implicit latent inference over a pre-existing mixture of experts @xcite . perhaps most importantly, while instruction tuning explicitly teaches a model to predict a task given it's context, it may also benefit in-context learning via multi-task transfer from training on hundreds of supervised tasks jointly. in this paper, we seek to better understand instruction tuning from the perspective of multi-task transfer. put simply, instruction-tuning involves continued training of a language model on supervised examples drawn from a variety of tasks. however, a common challenge when attempting to fit a single model to multiple tasks is interference between tasks during optimization, which can lead to models which generalize worse than single task models @xcite . despite these challenges inherent to multi-task learning, which are exacerbated with the number of tasks, the ability of an instruction-tuned model to perform in-context learning-learning the task to predict from examples or instructions provided in the context-improves with the number of tasks seen during training, because the model sees more demonstrations of context @xcite . we aim to characterize the extent to which multi-task transfer dynamics influence the resulting instruction-tuned model, and whether improving multi-task transfer can yield benefits to the transfer and in-context learning (icl) performance of instruction-tuned models towards unseen tasks. ## the trajectory of icl & forgetting in §4 we focus on improving multi-task transfer by explicitly attempting to address task imbalances and balance generalization across training tasks. in this section, we instead focus on improving model transfer by mitigating the catastrophic forgetting we observe in §3. more specifically, we ask can early-stopping and continual learning improve icl generalization by mitigating the effects of catastrophic forgetting on model transfer?