M ulti I nstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning (2023)

Abstract: Instruction tuning, a new learning paradigm that fine-tunes pre-trained language models on tasks specified through instructions, has shown promising zero-shot performance on various natural language processing tasks. However, it has yet to be explored for vision and multimodal tasks. In this work, we introduce MultiInstruct, the first multimodal instruction tuning benchmark dataset that consists of 62 diverse multimodal tasks in a unified seq-to-seq format covering 10 broad categories. The tasks are derived from 21 existing open-source datasets and each task is equipped with 5 expert-written instructions. We take OFA as the base pre-trained model for multimodal instruction tuning, and to further improve its zero-shot performance, we explore multiple transfer learning strategies to leverage the large-scale Natural Instructions dataset. Experimental results demonstrate strong zero-shot performance on various unseen multimodal tasks and the benefit of transfer learning from a text-only instruction dataset. We also design a new evaluation metric – Sensitivity, to evaluate how sensitive the model is to the variety of instructions. Our results indicate that fine-tuning the model on a diverse set of tasks and instructions leads to a reduced sensitivity to variations in instructions for each task.

## introduction with the advances in large-scale pre-trained language models (plms), recent studies have explored various efficient learning paradigms @xcite @xcite to generalize plms to new tasks without task-specific tuning. among these, instruction tuning @xcite has achieved significant success in zero-shot learning on natural language processing tasks. by fine-tuning a plm on tasks described through instructions, instruction tuning allows the model to learn to understand and follow the instructions to perform predictions on unseen tasks. recent advancement in multimodal pretraining @xcite @xcite has shown the potential of jointly interpreting text and images in a shared semantic space, which further leads us to ask: can the instruction tuning be leveraged to improve the generalizability of vision-language pretrained models on multi-modal and vision tasks? in this work, we propose multiinstruct, the first benchmark dataset for multimodal instruction tuning with 62 diverse tasks from 10 broad categories, including visual question answering @xcite , commonsense reasoning @xcite , visual relationship understanding @xcite and so on. we equipped each task with 5 instructions that are written by two experts in natural language processing. as shown in figure 1 , we formulate all the tasks into a unified sequence-to-sequence format in which the input text, images, instructions, and bounding boxes are represented in the same token space. we use ofa @xcite foot_0 , a unified model that is pre-trained on a diverse set of multimodal and unimodal tasks in a single transformerbased sequence-to-sequence framework, as the base pre-trained multimodal language model, and fine-tune it on multiinstruct. to utilize natu-ral instructions @xcite , a largescale text-only instruction tuning dataset, we further explore two transfer learning strategies, in- grounded caption text localization referring expression selection output: blue and white tennis racquet input: generate a caption @xmath0. input: select the region that contains the text "den". options: <@xmath1: select the region of the object described by "a blue train in the front.". options: <@xmath2 output: <@xmath3 output: <@xmath4 output: the question is irrelevant to the image ## experimental setup evaluation metrics we report the accuracy for classification tasks and rouge-l @xcite for all generation tasks. for the region classification task, we compute the intersection over union (iou) between the generated region and all regions in the options, select the option with the highest iou as the prediction, and compute accuracy based on this prediction. if the predicted region has no intersection with any of the regions in the options, we treat this prediction as incorrect. for classification tasks where the answer is not a single-word binary classification, we also report rouge-l scores following @xcite , which treats all tasks as text generation problems. for each task, we conduct five experiments by evaluating the model using one of the five instructions in each experiment. we re-port the mean and maximum performance and the standard deviation of the performance across all five experiments. we also compute the aggregated performance for each model based on the mean of the model's performance on all multimodal and nlp unseen tasks. we use rouge-l as the evaluation metric for most tasks and accuracy for tasks that only have accuracy as a metric. in addition, as instruction tuning mainly relies on the instructions to guide the model to perform prediction on various unseen multimodal tasks, we further propose to evaluate how sensitive the model is to the variety of human-written instructions in the same task, which has not been discussed in previous instruction tuning studies but is necessary to understand the effectiveness of instruction tuning. we thus further design a new metric as follows: sensitivity refers to the model's capability of consistently producing the same results, regardless of slight variations in the wording of instructions, as long as the intended task remains the same. specifically, for each task @xmath5, given its associated instances with task instructions: d @xmath6, x t j , y t j ) ∈ i t × x t × y t } n @xmath7, we formally define sensitivity as: ## conclusion we present a new large-scale multi-modal instruction tuning benchmark dataset -multiinstruct, which covers a wide variety of vision and multimodal tasks while each task is associated with multiple expert-written instructions. by finetuning ofa @xcite , a recently state-ofthe-art multimodal pre-trained language model, on multiinstruct with instruction tuning, its zeroshot performance on various unseen multimodal tasks is significantly improved. we also explore several transfer learning techniques to leverage the much larger text-only natural instructions dataset and demonstrate its benefit. moreover, we design a new evaluation metric sensitivity to assess the model's sensitivity towards the variations in the wording of instructions. results show that the model becomes less sensitive to these variations after being fine-tuned on a variety of tasks and instructions.