Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks (2023)

Abstract: Instruction tuning (IT) achieves impressive zero-shot generalization results by training large language models (LLMs) on a massive amount of diverse tasks with instructions. However, how to select new tasks to improve the performance and generalizability of IT models remains an open question. Training on all existing tasks is impractical due to prohibiting computation requirements, and randomly selecting tasks can lead to suboptimal performance. In this work, we propose active instruction tuning based on prompt uncertainty, a novel framework to identify informative tasks, and then actively tune the models on the selected tasks. We represent the informativeness of new tasks with the disagreement of the current model outputs over perturbed prompts. Our experiments on NIV2 and Self-Instruct datasets demonstrate that our method consistently outperforms other baseline strategies for task selection, achieving better out-of-distribution generalization with fewer training tasks. Additionally, we introduce a task map that categorizes and diagnoses tasks based on prompt uncertainty and prediction probability. We discover that training on ambiguous (prompt-uncertain) tasks improves generalization while training on difficult (prompt-certain and low-probability) tasks offers no benefit, underscoring the importance of task selection for instruction tuning.

## introduction recently, instruction tuning has shown great success in improving large language models' crosstask generalizability. when training large language models (llm) with a wide range of tasks with instructions, models like t0 @xcite , flan @xcite , tk-instruct @xcite , instruct-gpt @xcite , alpaca @xcite and vicuna @xcite given an instruction tuning (it) model with a sizeable existing task pool, we actively identify and select useful tasks from it and add them to the training task pool. after that, we train a new it model with the updated training task pool. by continuing this loop, we expect to improve the it model's cross-task generalization ability efficiently. the main challenge lies in identifying useful tasks, for which we propose to select prompt-sensitive tasks. data selection has been explored under active learning and multi-task learning frameworks. despite its prevalence, we argue that they are not applicable to task selection for instruction tuning. specifically, active learning methods have focused on selecting the most useful instances for a single task, using either uncertainty-based intuitions such as entropy @xcite , monte carlo dropout @xcite , or ensemble disagreement @xcite . however, these uncertainty measurements can only measure uncertainty at instance-level, and will become less effective when applied to task-level selections as the scales of uncertainty values are not comparable across tasks. in multi-task learning, previous research @xcite @xcite has explored measuring task usefulness by assessing its similarity to the target task. while these methods can enhance performance when aware of the target tasks, they are not suitable for instruction tuning, which aims to improve overall generalization to arbitrary unseen tasks. in this work, we introduce active instruction tuning, a framework that aims to actively identify informative new tasks for an it model to continuously improve its cross-task generalization ability (refer to figure 1 ). while being related to active learning, our task is more challenging. unlike active learning, which focuses on improving performance on a single target task by identifying useful instances, our goal is to identify tasks that enhance overall generalization, a novel concept not explored in previous al research. to identify informative new tasks for active instruction tuning, we propose prompt uncertainty (refer to figure 2 ), a novel task-level uncertainty metric that measures the sensitivity of an it model against instruction perturbations for a task. specifically, with a task instruction and a few unlabeled instances, we assess the disagreement of model predictions against original and perturbed prompts on multiple instances to obtain an average disagreement score. we then select and train the model with the most prompt-uncertain tasks to enhance the overall cross-task generalization ability. since this uncertainty method does not require labeled instances for a task, we can also apply prompt uncertainty to determine novel tasks to manually annotate if needed. we further explore using prompt uncertainty to understand task characteristics and diagnose potential issues. motivated by data map @xcite , which utilizes instance-level training dynamics to categorize and diagnose data quality, we propose task map, the first task diagnosing method that categorizes tasks based on prompt uncertainty and prediction probability. based on the task map, we categorize tasks into ambiguous, easy and difficult, inspired by prior in-context learning research @xcite to facilitate analysis. we conduct experiments on two instruction tuning setting: tk-instruct models with niv2 dataset, which generalize to unseen tasks, and alpaca models with self-instruct dataset, which generalize to unseen instructions, following our categorization in @xcite . results show that our active instruction tuning method consistently outperforms baseline methods (random sampling, generation perplexity) for both instruction tuning setting, demonstrating the effectiveness of our approach. moreover, we discover that while instruction tuning with ambiguous tasks can improve generalization effectively, difficult tasks offers no benefit, underscoring the importance of task selection in instruction tuning. our contributions can be summarized as follows: ## experiment setting in this work, we experiment with two well-known it datasets: niv2 and self-instruct @xcite . niv2 is the largest it dataset with 1600+ cross-lingual tasks. it focuses on improving model generalization to unseen tasks, while self-instruct is used to train the alpaca model @xcite and aims to enhance model instruction following ability, following the categorization in prior work @xcite setting and comparison, please see ## conclusion we propose active instruction tuning with prompt uncertainty, a framework to enhance the generalization ability of the it model in large-scale instruction tuning. our experiments on niv2 and self-instruct datasets demonstrate that training on prompt uncertain tasks consistently outperforms random sampling and other uncertainty baselines, highlighting the effectiveness of our approach. we also introduce task map, a tool that categorizes tasks based on prompt uncertainty and prediction probability, revealing that while training on ambiguous tasks improves generalization, some difficult tasks offer no benefit. these findings motivate future investigations into prompt uncertainty and task selection strategies for better understanding cross-task generalization and instruction tuning.