Instruction Matters: A Simple yet Effective Task Selection for Optimized Instruction Tuning of Specific Tasks (2024)

Abstract: Instruction tuning has been proven effective in enhancing zero-shot generalization across various tasks and in improving the performance of specific tasks. For task-specific improvements, strategically selecting and training on related tasks that provide meaningful supervision is crucial, as this approach enhances efficiency and prevents performance degradation from learning irrelevant tasks. In this light, we introduce a simple yet effective task selection method that leverages instruction information alone to identify relevant tasks, optimizing instruction tuning for specific tasks. Our method is significantly more efficient than traditional approaches, which require complex measurements of pairwise transferability between tasks or the creation of data samples for the target task. Additionally, by aligning the model with the unique instructional template style of the meta-dataset, we enhance its ability to granularly discern relevant tasks, leading to improved overall performance. Experimental results demonstrate that training on a small set of tasks, chosen solely based on the instructions, results in substantial improvements in performance on benchmarks such as P3, Big-Bench, NIV2, and Big-Bench Hard. Significantly, these improvements surpass those achieved by prior task selection methods, highlighting the superiority of our approach.

## introduction recently, instruction tuning has gained attention as an innovative approach for improving zero-shot performance @xcite @xcite @xcite . this method offers the advantage of improving a model's generalization capabilities to unseen tasks by training on diverse tasks accompanied by instructions. the robustness 2022; @xcite . however, the former approach, which measures pairwise transfer for every task, can be exceedingly time-consuming and compute-intensive. moreover, the latter approach still necessitates the construction of data for unseen target tasks, which not only introduces additional burdens but also does not fully align with the fundamental goal of instruction tuning, which is to enhance zero-shot capabilities. in this study, we explore a straightforward yet effective method for choosing relevant tasks for optimized instruction tuning. we focus on the feature of instruction tuning, where the instructions foot_2 define the characteristics of each task. building upon this, we introduce an instruction-based task selector (insta) that leverages instruction similarity scores to determine task relevance and select informative tasks for training. through experiments, we discover that this simplified instruction-only selection method adequately identifies related tasks and significantly improves performance. moreover, by further aligning insta through training on the instruction style and format of a meta-dataset, it is able to closely understand the instructional nuances, achieving significant performance improvements. a notable finding from our study is that task selection based exclusively on instructions surpasses previous sample-based methods @xcite @xcite that employ both instructions and instances. moreover, instruction information alone shows a high correlation with task similarity as determined by complex pairwise transfer methods @xcite and even achieves slightly better average zero-shot performance. this indicates that the instruction-based approach for selecting related tasks is not only easily applicable but also highly effective. in summary, our contributions are as follows: ## instruction-based task selector (insta) to enhance the zero-shot capability of llms for specific target tasks, we select informative tasks that positively impact the performance of the target task. our task selection method exclusively relies on instruction information to assess the relevance between tasks, which is efficient as it removes the necessity of measuring correlations between all training and evaluation task pairs. furthermore, unlike previous methods, our approach has the advantage of easy applicability with just the instruction (task description) of the target task, without the need for constructing data samples for the target task. ## further analysis for further analysis, we conduct experiments to compare instruction-based versus data samplebased task selection, examine the effects of instruction refinement in the p3 dataset, and assess the impacts of varying the number of related tasks selected (k). additionally, changes in task selection performance based on the number of instructions used for alignment, as well as experiments on learning different styles of meta-datasets and their impact on performance, are detailed in appendices k and l. ## conclusion in this study, we discover that selecting informative tasks for instruction tuning can be effectively achieved by exclusively using task instructions. our experiments reveal that this method, particularly when aligned with the meta-dataset's instructions, surpasses traditional methods which depend on data samples to determine task relevance. beyond its robust performance, our approach's most significant advantage is its simplicity: as long as the instruction (task description) of the target task can be described, our method can be applied. this approach marks a significant shift away from traditional methods that require exhaustive pairwise comparisons or the labor-intensive creation of data samples for new tasks. by adopting an instructiononly strategy, our method simplifies and automates the task selection process for instruction tuning, providing a more efficient and practical approach to developing models in real-world scenarios.