How Many Data Samples is an Additional Instruction Worth? (2023)

Abstract: Recently introduced instruction-paradigm empowers non-expert users to leverage NLP resources by defining a new task in natural language. Instruction-tuned models have significantly outperformed multitask learning models (without instruction); however they are far from state-of-the-art task-specific models. Conventional approaches to improve model performance via creating datasets with large number of task instances or architectural changes in the model may not be feasible for non-expert users. However, they can write alternate instructions to represent an instruction task. Is Instruction-augmentation helpful? We augment a subset of tasks in the expanded version of NATURAL INSTRUCTIONS with additional instructions and find that it significantly improves model performance (up to 35%), especially in the low-data regime. Our results indicate that an additional instruction can be equivalent to ~200 data samples on average across tasks.

## introduction large-scale benchmarks such as imagenet @xcite , squad @xcite and architectural development in models such as cnns @xcite and transformers @xcite have propelled our progress in deep learning. however, creating high-quality benchmarks by controlling its artifacts @xcite , developing new models, and training them is hard for non-expert users. recently introduced instructionparadigm empowers non-expert users, practitioners, and domain experts in other fields to leverage nlp resources @xcite as they now can describe their tasks in natural language without requiring to create task-specific datasets or developing models 2 . even though the instruction paradigm has led to the development of models that significantly outperform multitasking baselines, model performance has remained far behind the supervised learning model trained with task-specific data @xcite . non-expert users can write multiple instructions per task each of which covers multiple perspectives spanning over a variety of linguistic features; many of these can be created automatically by replacing certain words with their synonyms without changing the overall semantics of instruction. can the relatively inexpensive process of instruction augmentation improve the model's performance in the instruction-paradigm, similar to the role data-augmentation has played conventionally in machine learning @xcite ? instructionparadigm is pivotal where it is expensive or infeasible to gather training data. how effective is instruction augmentation in low-data regimes? multi-variant instructions (original + augmented instructions) also can help evaluate the robustness of instruction-following models to respond to variant instructions. this is similar to the model robustness evaluation @xcite that is done by creating variant data instances. multi-variant instruction-based setup will also help gauge the true potential of instruction-following systems since in a real-world setting, users can write task instructions in many different ways. the expanded version of natural instruc-tions @xcite foot_0 provides a rich collection of the diverse category of tasks that covers a variety of reasoning skills, domains, and languages. this constantly evolving benchmark is growing in size with respect to time. we take 426 tasks 4 and creates variant instructions for each task. in natural instructions, the number of instances was limited to 6500 to reduce massive data imbalance, we leverage the remaining instances of source datasets in constructing instances of our variant instruction tasks. we experiment with 3 types of learning scenarios (i) taskspecific (ts), (ii) multi-task (mt), and (iii) crosstask (ct) and observe that instruction augmented models outperform their single-instruction counterparts by 17%, 11%, and 11%, respectively when averaged over all experiments across the evaluation tasks. interestingly, instruction augmentation is more effective on the low-data regime (average across 1%, 5%, and 10% data) as we see a performance gain of 26%, 16%, and 11% in ts, mt, and ct settings, respectively. we also quantify the contribution of each of the additional instructions and find that an additional instruction can be equivalent to "200 data samples on average across tasks. ## multi-variant instruction dataset we construct a multi-variant instruction dataset on top of various tasks in natural instructions. in total, our dataset has 426 different nlp tasks; each of which contains multi-variant instructions. ## conclusion we introduced instruction augmentation to improve existing lms in terms of improving performance and usability to non-expert users. to this extent, we created multi-variant instructions for 426 nlp tasks. our experiment results show that instruction augmentation improves model performance in task-specific, multi-task and cross-task learning paradigms. we find that instruction augmentation is more effective in low-data regime. our results further indicate that an additional instruction can be equivalent to "200 instances on an average. we hope our work will bring more attention to developing unconventional techniques (beyond dataset creation and model training) to empower non-expert users to leverage nlp resources and teach a task without having domain knowledge.