Paper: Understanding Multimodal Instruction Format for In-context Learning (ICLR-2024)

Reviewer comment (Novelty):
This work mainly focuses on instruction formats, lacks innovation, and fixed instruction formats are difficult to generalize to open-domain tasks.

Author response:
**A1:** We acknowledge your concerns regarding the generalization of our method to open-domain tasks. Here, we categorize the open-domain tasks you mentioned into two types. The first type belongs to traditional vision-language tasks, such as open-domain Visual Question Answering (VQA) tasks like GQA [16], VQAv2 [17], etc. Our proposed UMIT can generalize to these tasks, which has been demonstrated by our results shown in Table 6.
The second type is real-world tasks. We must acknowledge that, since real-world instructions are diverse and challenging to describe in terms of task definition, our framework currently has limitations when extending to such tasks. However, we contend that, at present, existing MLLMs other than GPT-4V lack in-context learning abilities for real-world tasks. Simultaneously, there is a lack of a reasonable benchmark to effectively evaluate the in-context learning ability of MLLMs in real-world tasks and provide quantifiable results. Therefore, this should be considered a common limitation in the current field of multimodal large language models, rather than being exclusive to our work. A detailed discussion can be found in our comments to reviewer 4eXe in **A2**.

Reviewer comment (Experiments):
When testing, does UMIT require using ChatGPT to obtain the task definition for each new sample? This can result in significant inference costs for the unseen task.

Author response:
**A2:** No, during the testing stage, UMIT only requires using ChatGPT to obtain the task definition for each task rather than each sample. As demonstrated in Table 12 and Table 13, we showcase the task definitions we obtained for tasks from the MME benchmark and SEED Bench using ChatGPT before the inference stage. This process is handled before inference, so it does not add to the inference costs.

Reviewer comment (Writing):
The description of the details of UMIT, especially the use of symbols, is somewhat confusing. For example, what does the text encoder encode in exemplar sampling? And where does X_{instruct}^{i} come from in section 2.5?

Author response:
**A3:** Sorry for this confusion. We will revise the paper to clarify them. 
+ we use text embeddings of both question X_q and answer X_a to retrieve in-context exemplars in training stage. This ensures that we can still retrieve similar in-context examples in tasks with specific questions other than VQA tasks. And for all evaluations, we randomly select in-context exemplars in testing stage.
+ All X_{instruct}^{i} are derived from annotations in the original datasets we collected. In our response **A1** to **reviewer 4eXe**, we provided a detailed list of three instances from three distinct multimodal instruction datasets. We hope that reviewing these instances will help clarify this confusion.

Reviewer comment (Evaluation):
The experimental results on SEED-Bench show a significant improvement when training OpenFlamingo and Otter on the data collected by the authors. In contrast, the gains from changing the format seem less pronounced. This raises a question of whether the role of data diversity is much greater than the task definition proposed by the author.

Author response:
**A4:** We highly agree with your perspective. At least for the present, the improvement brought about by increasing task diversity on traditional vision-language metrics is indeed irreplaceable. This is also confirmed by LLaVA 1.5 [18], and in the NLP community, it has also been proven, as seen in works like Flan-2022 [19], opt-iml [20], etc. The purpose of introducing "task definition" is also to effectively merge existing multimodal instruction datasets using a unified format. This is done to enhance task diversity while, as much as possible, preserving the ability for in-context learning without compromising zero-shot performance.

Reviewer comment (Evaluation):
Some experimental results on VizWiz contradict the author's conclusions and should be analyzed in more detail. For example, in Table 4, "DEIQA (F3) mixed" performs worse than "DEIQA (F3) random". And in Table 5, "vqa+diff" performs worse than "vqa+same".

Author response:
**A5:** We apologize for the insufficient analysis, and we will revise the paper to provide a clearer analysis of these results. The performance drop from "random" to "mixed" might be attributed to the random selection of in-context examples during the testing stage, which could introduce a certain level of bias. Additionally, on the other two datasets, HatefulMemes and ISEKAI, the performance of "DEIQA (F3) mixed" and "DEIQA (F3) random" is comparable. Therefore, the bias introduced by in-context examples might result in a slightly inferior performance for "mixed" compared to "random." To investigate this phenomenon, we conducted three random seed selections for in-context examples and obtained the following results:

| **Method**           | **0** | **2** | **4** | **8** |
|----------------------|-------|-------|-------|-------|
| DEIQA + Random (avg) | **27.15** ±0.14 | 29.53 ±1.06 | 30.66 ±0.86 | 31.21 ±0.67 |
| DEIQA + Mixed (avg)  | 27.03 ±0.06 | **29.66** ±0.98 | **31.02** ±0.96 | **31.87** ±0.29

Reviewer comment (Experiments):
Do other ICL methods harm zero-shot performance?

Author response:
**A7:**  In our response **A1** to **reviewer 4eXe**, we provided a detailed list of three instances from three distinct multimodal instruction datasets, including Otter dataset, MMICL dataset, MultiInstruct dataset. We hope that reviewing these instances will help clarify this confusion.

Reviewer comment (Novelty):
The contribution and novelty of this work is insufficient. Adding a carefully designed task definition with minor improvements (0.3 in Tab3) is not that significant in terms of technical contributions or scientific findings.

Author response:
Thank you for your careful examination of our table. In fact, the UMIT framework is designed to efficiently merge different multimodal instruction datasets with various tasks using a unified format, preserving the model's in-context learning ability on traditional vision-language tasks to the greatest extent possible without damaging the zero-shot performance of MLLMs. 

We contend that multimodal instruction datasets are currently in a nascent stage of development, with significant limitations in terms of task diversity. Currently, the multimodal instruction dataset with the highest number of tasks is vision-flan [13], boasting over 200 tasks. However, this is still considerably fewer than instruction datasets in the NLP community, such as Flan-2022 [14] (1836 tasks) and Ni-v2 [15] (1616 tasks). Therefore, to rapidly increase task diversity, we can merge existing multimodal instruction datasets. Designing a unified format is what we consider the foremost requirement for merging these datasets, and this is the significance behind the birth of the UMIT framework.

Regarding Table 3, our primary focus has been to investigate the impact of the Unified Multimodal Instruction Format (UMIT) on multimodal in-context learning. In Table 3, our format demonstrates a performance improvement of 1.2 through in-context learning, while the otter-style format yields a more modest improvement of 0.5. Although the OpenFlamingo-style format exhibits a stronger in-context learning capability, with an improvement of 2.5, this is due to its lack of instructions. Consequently, it minimizes the gap between fine-tuning and pre-training data format, preserving in-context learning to the greatest extent. However, the consequence is a significant impact on its zero-shot performance. Therefore, we consider our format to be the optimal compromise.

Moreover, given that SEED Bench comprises 15,425 examples, we argue that both the 0.7 improvement in few-shot performance compared to the Otter-style format and the remarkable 3.2 improvement in zero-shot performance compared to the OpenFlamingo-style format are highly significant and noteworthy.

Reviewer comment (Novelty):
The literature review in Sec2.1 is not correct, e.g., "Moreover, Otter (Li et al., 2023b) and MMICL (Zhao et al., 2023b) further introduced". Otter was before Qwen-VL.

Author response:
We apologize for this error. It occurred because we wrote in the order of the increasing complexity of their instruction format components. In the Qwen-VL format, there is a lack of instructions, while Otter includes instructions, so we placed Qwen-VL first. We will revise the article to correct this error in Section 2.1. Part of the revised text is as follows:
> Unlike the format utilized by Qwen-VL, Otter and MMICL incorporated the instance-level instruction component into the prompt format design for in-context instruction tuning.

Reviewer comment (Novelty):
The main contribution here seems to be the new "task definition" component of the prompt that precedes the in-context examples, which does not seem particularly novel, and looking at the examples in Figure 1, I don't really see what information they provide that is not provided in the instance instruction itself -- the task definition just seems like a more verbose form of the instance-level instruction.

Author response:
**Q1: The difference between the extra information provided about "task definition" and "instance-level instruction".**

**A1**: We understand your concerns about whether adding the "task definition" component can enhance information beyond that provided by "instance-level instruction.". Firstly, we would like to illustrate the differences between the three components: "task definition," "instance-level instruction," and "question," by presenting an instance from one of the three different multimodal instruction datasets:
1. **Otter-LA dataset**. This dataset lacks the "task definition" component. And the majority of this dataset is derived from real-world tasks, so we consider the "instruction" and the "question" as the same component. 
```
### Otter-LA dataset ###

[component 1] Task Definition: None

[component 2] In-context Exemplars: 
Instruction 1: None
Question1: What skill set might someone need to perform such a frisbee trick?
Answer1: To perform the frisbee trick shown in the image, where the man is passing a frisbee between or underneath his legs, a person would need a combination of skills...
Instruction 2: None
Question2: ...
Answer2: ...
...

[component 3] Instruction: None (you can consider the "question" as an "instance-level instruction".)
[component 4] Question: What skills or techniques might the man need to successfully play with the frisbee?
[component 5] Answer: To successfully play with the frisbee, the man needs various skills and techniques such as hand-eye coordination, spatial awareness, and understanding the aerodynamics...
```

2. **MMICL dataset**: This dataset also lacks the "task definition" component. However, the majority of its examples come from traditional vision-language tasks, such as OK-VQA, COCO-caption, etc. Therefore, we can distinguish its "instruction" component from the "question" component.
```
### MMICL dataset (OK-VQA) ###

[component 1] Task Definition: None

[component 2] In-context Exemplars: 
Instruction 1: Make sure your answers are based on the information presented in the image 0: <image0>图.
Question 1: What is the purpose of the red decorations in this photo? 
Answer 1: lighting
Instruction 2: ...
Question 2: ...
Answer 2: ...

[component 3] Instruction: Carefully examine image 2 labeled <image2>图 before answering the question.
[component 4] Question: Name the type of plant this is?
[component 5] Answer: vine
```

3. **MultiInstruct dataset**: This dataset encompasses 62 traditional vision-language tasks, with 5 task-level instructions manually crafted for each task, referred to as "task definition." Therefore, it does not require additional "instance-level instruction." However, it lacks the "in-context exemplars" component.
```
### MultiInstruct dataset (OK-VQA) ###

[component 1] Task Definition: In this task, you will be asked a question about image. However, in order to answer this question, you need knoweldge outside of this image. 

[component 2] In-context Exemplars: None

[component 3] Instruction: None (you can consider the "task definition" as an "task-level instruction".)
[component 4] Question: Name the type of plant this is?
[component 5] Answer: vine
```

Therefore, in our definition, the "task definition" component contains background information about the task, while the "instruction" component only guides the model to use the image content to answer relevant questions, without containing specific task information. Thus, with the addition of "task definition," we can even consider "instance-level instruction" and "question" as the same component.

In fact, the UMIT framework is designed to efficiently merge different multimodal instruction datasets with various tasks using a unified format, preserving the model's in-context learning ability on traditional vision-language tasks to the greatest extent possible without damaging the zero-shot performance of MLLMs. 

We contend that multimodal instruction datasets are currently in a nascent stage of development, with significant limitations in terms of task diversity. Currently, the multimodal instruction dataset with the highest number of tasks is vision-flan [1], boasting over 200 tasks. However, this is still considerably fewer than instruction datasets in the NLP community, such as Flan-2022 [2] (1836 tasks) and Ni-v2 [3] (1616 tasks). Therefore, to rapidly increase task diversity, we can consolidate existing multimodal instruction datasets. Designing a unified format is what we consider the foremost requirement for merging these datasets, and this is the significance behind the birth of the UMIT framework.

Reviewer comment (Theory):
The retrieval augmentation in Section 2.4 requires us to have a reasonably large database of training examples to choose from, which may enhance task performance on benchmarks but defeats the purpose of in-context learning in the real world.

Author response:
**Q2: The retrieval augmentation in Section 2.4 requires us to have a reasonably large database of training examples to choose from, which may enhance task performance on benchmarks but defeats the purpose of in-context learning in the real world.**

**A1:** Firstly, we think that our work is highly meaningful. Currently, in the multimodal field, there are numerous instruction datasets such as LLaVA, Otter, MultiInstruct, Vision-Flan, etc. However, the tasks included in these datasets are very limited. For example, even Vision-Flan only has 200 tasks, which still represents a significant gap compared to instruction datasets in the NLP community, such as Flan-2022 with over 1800 tasks. Additionally, after visual instruction tuning, the in-context learning capability of MLLMs tends to decline, as evident in Table 2 and Table 3. Therefore, based on these two problems, we propose a unified multimodal instruction format to merge existing datasets, increase task diversity, and, at the same time, maximize the retention of in-context learning ability on traditional visual language tasks. 

We strongly agree with your perspective on the significant value of in-context learning in the real world, surpassing its importance in traditional vision-language tasks. However, the focus of our paper still lies in the latter, aiming to enhance the in-context learning ability of MLLMs on traditional vision-language tasks through retrieval methods. There are two reasons for this:
+ Firstly, we argue that, except for GPT-4V, all current MLLMs lack the capability to achieve in-context learning in the real world. For instance, MLLMs like Flamingo [4], OpenFlamingo [5], Kosmos-2 [6], Emu [7], Otter [8] among others, merely showcase their few-shot performance on traditional vision-language tasks. However, there are no other MLLMs demonstrating real-world in-context learning examples as illustrated in Figures 8 — 13 in GPT-4V technical report [9]. 
+ Secondly, We think that there is currently a lack of a reasonable evaluation method and dataset to effectively measure MLLMs' in-context learning ability in the real world. For example, in Figures 8 to 10 of the GPT-4V technical report [9], GPT-4V accurately judges the speed range of a dashboard through in-context learning, demonstrating the utility of in-context learning. However, we face challenges in quantifying it effectively.

In summary, we believe that exploring the in-context learning ability of MLLMs in the real world still requires further investigation. In our paper, we can only focus on the performance improvement on traditional VL metrics brought about by constructing a unified multimodal instruction format using retrieval method.

Reviewer comment (Writing):
In equation 5, is the text embedding E_text computed using the question X_q, answer X_a or both? this is not made clear.

Author response:
**Q4: In equation 5, is the text embedding E_text computed using the question X_q, answer X_a or both?**

**A4:** Thank you for your comments! We are sorry that we didn't make that clear in Section 2.3. We’ve revised the content of Section 2.3. 
In fact, we use text embeddings of both question X_q and answer X_a to retrieve in-context exemplars in training stage. This ensures that we can still retrieve similar in-context examples in tasks with specific questions other than VQA tasks. However, for all evaluations, we randomly select in-context exemplars in testing stage and our method still can achieve promising performances.