Differentiable Instruction Optimization for Cross-Task Generalization (2023)

Abstract: Instruction tuning has been attracting much attention to achieve generalization ability across a wide variety of tasks. Although various types of instructions have been manually created for instruction tuning, it is still unclear what kind of instruction is optimal to obtain cross-task generalization ability. This work presents instruction optimization, which optimizes training instructions with respect to generalization ability. Rather than manually tuning instructions, we introduce learnable instructions and optimize them with gradient descent by leveraging bilevel optimization. Experimental results show that the learned instruction enhances the diversity of instructions and improves the generalization ability compared to using only manually created instructions.

## introduction recently, significant progress has been made in developing models that can generalize to arbitrary tasks by following natural language descriptions @xcite . instruction tuning has been a region of interest as a training technique to obtain such generalization ability @xcite @xcite . by finetuning pretrained language models on a variety of tasks with their instructions, models can generalize to arbitrary tasks unseen during training. many previous studies witnessed the effectiveness of instruction tuning @xcite @xcite . various instructions have been created for instruction tuning, such as task name, task definition, positive/negative exemplars of a task, explanations of why each positive/negative exemplar is correct/incorrect, etc. however, @xcite ; @xcite showed that the definition and positive exemplars of tasks are sufficient for instruction tuning, and the effect of adding other types of instruction is negligible or sometimes has a answer the sentiment of the text. this film is fun. translate the text into japanese. negative impact on the generalization performance. seeking an optimal instruction for cross-task generalization is an important issue for instruction tuning, while it requires much human effort (100+ researchers have participated in previous studies). furthermore, human-interpretable instructions are not necessarily optimal for obtaining cross-task generalization ability. against this background, we propose instruction optimization, which introduces learnable instructions and optimizes them w.r.t. the cross-task generalization ability. as shown in figure 1 , a model θ is optimized to maximize the performance on meta-train tasks following learnable instructions. by contrast, learnable instructions ϕ are trained to maximize the meta-test performance of the trained model θ * (ϕ). this optimization is called bilevel optimization and is frequently used in hyperparameter optimization @xcite , meta-learning @xcite , and neural architecture search @xcite . we regard training instructions as a special type of hyper-parameter and optimize them with gradient descent by relaxing the search space to be continuous. to create learnable instructions, we propose two methods: instruction embedder, which generates the embeddings of instructions, and instruction extractor, which selects an optimal task exemplar. recently, prompt engineering has drawn attention to seek the optimal prompt to achieve a task @xcite . some work studies continuous prompts that perform prompting in the embedding space of tokens @xcite , whereas others retrieve optimal exemplars as a testing prompt for in-context learning @xcite . our instruction embedder and instruction extractor follow the idea of continuous prompts and prompt retrievers, respectively. whereas previous work optimizes prompts to solve an individual task on the test, our study differs in the target and aim of optimization. we optimize the training prompts to maximize the cross-task generalization ability of the trained model. in the experiment, we confirmed that the instruction extractor successfully extracted appropriate instruction, providing proof of concept. regarding the comparison with instruction tuning, the instruction embedder enhances the diversity of instructions and improves the generalization ability compared to using only manually created instructions. in contrast, the instruction extractor does not contributes to the performance gain, which shows that using the same task exemplar across instances is unexpectedly preferable for cross-task generalization. this study provides a basis for exploring the optimal instructions for instruction tuning. ## preliminaries instruction tuning trains a model θ to minimize the training loss defined in eq. ( 1 ): ## instruction optimization instruction optimization splits training tasks t train into two sets: meta-train tasks t meta-train and meta-test tasks t meta-test . subsequently, a model θ is trained to minimize the inner loss on meta-train tasks following learnable instructions i ϕ in eq. ( 2 ). this optimization is called bilevel optimization and is commonly used in hyperparameter optimization. figure 2: outline of instruction embedder and instruction extractor. instruction tuning uses a manually created instruction or randomly selected exemplar as training instruction. in contrast, instruction embedder introduces the learnable embeddings of instruction, while instruction extractor selects an optimal exemplar as training instruction. note that we use the manually created instruction i t to measure the meta-test performance because we aim to develop a model that can accept arbitrary human-created instructions. ## conclusion this study presents instruction optimization, which optimizes training instructions concerning generalization ability. the experimental results showed that our instruction extractor successfully extracted appropriate instruction, providing proof of concept. regarding the comparison with instruction tuning, the instruction embedder enhanced the diversity of instructions and improved the generalization ability than using only manually created instructions. in contrast, the instruction extractor did not contribute to the performance gain because using the same task exemplar across instances is unexpectedly preferable for cross-task generalization. this study provides a basis for exploring the optimal instructions for instruction tuning.