Paper: Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models (ICLR-2024)

Reviewer comment (Experiments):
It only consider two images per context, which could be too structure with little diversity.

Author response:
As shown in Table 1, our dataset spans one to four images across one or two dialogue turns and is not limited to two images. Additionally, the model can generalize to scenarios with more than four images or two turns. Additionally, **we include examples with four or five images in Appendix F of the revised manuscript to demonstrate this expanded capability.** We acknowledge your point on diversifying conversation patterns and plan to introduce additional templates and user-initiated conversation structures to expand the range of dialogue scenarios.

Reviewer comment (Novelty):
The SparklesChat model is not novel in that it is just an instruction tuned miniGPT-4. It could be removed from contributions.

Author response:
Our contribution to the model is the open-source SparklesChat to support chats across multiple images and dialogue turns. This capability has been a frequent request in issue trackers that remain unresolved since the release of popular multimodal instruction-following models such as MiniGPT-4 and LLaVA (see [issues 1](https://github.com/haotian-liu/LLaVA/issues/197), [2](https://github.com/Vision-CAIR/MiniGPT-4/issues/180), and [3](https://github.com/Vision-CAIR/MiniGPT-4/pull/232). We believe SparkelsChat is contributing to the community by pushing the boundaries of existing architectures to address novel challenges.

Reviewer comment (Experiments):
Each conversation seems to have a very typical pattern with two images as described in section 2. “In the first turn, the user initiates a reasonable and creative message regarding some images. In response, the assistant generates detailed answers that include comprehensive reasoning regarding the visual content. In the second turn, the user introduces a new image for further discussion, referencing both the new and previous images.”

Author response:
As shown in Table 1, our dataset spans one to four images across one or two dialogue turns and is not limited to two images. Additionally, the model can generalize to scenarios with more than four images or two turns. Additionally, **we include examples with four or five images in Appendix F of the revised manuscript to demonstrate this expanded capability.** We acknowledge your point on diversifying conversation patterns and plan to introduce additional templates and user-initiated conversation structures to expand the range of dialogue scenarios.

Reviewer comment (Experiments):
Only the miniGPT-4 is compared as a baseline.

Author response:
SparklesChat is built upon the MiniGPT-4 and adapted to accommodate multiple images due to its relevance and prominence in the field. We **extended our approach to more advanced models such as LLaVA-v1.5**to offer a broader view of our approach's effectiveness. LLaVA-v1.5 has improved its image input resolution from 224 pixels to 336 pixels. It is trained on a diverse dataset comprising about 665K instruction-following data. Additionally, LLaVA-v1.5* is fine-tuned on SparklesDialogue (about 6.5K) using the low-resource technique LoRA to conserve memory and time. The results are shown below and have **updated in the revised manuscript in Section 5.1 and Table 2.**

| Model        | Instruction Data| BISON | NLVR2 | SparklesEval |
|--------------|-----------------|-------|-------|-------|
| GPT-4        | -               | -     | -     | 9.26  |
| MiniGPT-4    | description     | 46.0% | 51.3% | 3.91  |
| MiniGPT-4*   | description     | 51.3% | 46.7% | 3.50  |
| LLaVA-v1.5   | Mixture (665K ) | 52.7% | 53.3% | 2.75  |
| LLaVA-v1.5*  | **+SparklesDialogue**| 65.3% | 56.7% | 7.93  |
| SparklesChat | description     | 52.0% | 48.0% | 3.06  |
| SparklesChat | reasoning       | 52.7% | 54.0% | 6.71  |
| SparklesChat | SparklesDialogue| 56.7% | 58.0% | 8.56  |

According to the results, despite LLaVA-v1.5's advantages of higher image resolution (336 vs. 224 pixels) and a larger training set (665K vs. 6.5K), SparklesChat significantly outperforms LLaVA-v1.5 in three evaluation sets involving multiple images. While LLaVA-v1.5 outperforms MiniGPT-4 on BISON and NLVR2, it shows weaker results on SparklesEval. This may be due to LLaVA-v1.5's training data primarily focusing on closed-set multimodal tasks such as VQA, TextCaps, and RefCOCO, while lacking in open-ended dialogue training. After fine-tuning with our SparklesDialogue, LLaVA-v1.5* not only significantly improved in open-ended dialogue tasks but enhanced traditional multimodal tasks. These results validate the adaptability of our method in unlocking chats across multiple images for multimodal instruction-following models with minimal additional training cost.

Reviewer comment (Experiments):
Why only the two datasets - BISON and NLVR2 are chosen? Is there any other dataset to use?

Author response:
NLVR and BISON were selected for their use of natural images and the feasibility of automatic evaluation with unambiguous answers (e.g., TRUE/FALSE, single-choice problems). Based on your feedback, we aim to include broader evaluation benchmarks, incorporating images from diverse domains (e.g., documents, fashion, cartoons) and a wider range of evaluation metrics beyond accuracy measures.

Reviewer comment (Evaluation):
SparklesDialogue contains subset SparklesDialogueVG, which was generated using GPT-4. The paper compares with performance of GPT-4 (method used to create the data set is also being evaluated on), while still performing worse although SparklesChat uses much richer image embedding.

Author response:
SparklesChat underperforms compared to GPT-4, despite using richer image representations, because GPT-4 operates based on detailed image annotations and the evaluation set was generated by GPT-4. This gives GPT-4 an inherent advantage in image understanding tasks, as it does not face the same challenges in interpreting images.

Reviewer comment (Novelty):
No contribution in terms of novelty architecture. Main contribution is in the data set.

Author response:
Thank you for pointing out that the dataset is our main contribution. Our contribution to the model is the open-source SparklesChat to support chats across multiple images and dialogue turns. This capability has been a frequent request in issue trackers that remain unresolved since the release of popular multimodal instruction-following models such as MiniGPT-4 and LLaVA (see [issues 1](https://github.com/haotian-liu/LLaVA/issues/197), [2](https://github.com/Vision-CAIR/MiniGPT-4/issues/180), and [3](https://github.com/Vision-CAIR/MiniGPT-4/pull/232)). We believe SparkelsChat is contributing to the community by pushing the boundaries of existing architectures to address novel challenges.

Reviewer comment (Experiments):
Only two turns per sample in the dataset. Longer sessions are probably more practical than more images per turn and limiting to just 2 turns. Dataset (that too, machine-generated) being the highlight of this paper, would have expected more.

Author response:
We initially created two-turn dialogues to validate our ideas in unlocking chats across multiple images and turns, which is adaptable to more turns, as shown in Figure 8. Additionally, we acknowledge the practicality of longer sessions and plan to expand the dataset accordingly. This can be achieved by incorporating more turns of conversation based on the current two-turn dialogues or exploring new scenarios for images and text from diverse domains (e.g., documents, fashion, cartoons).

Reviewer comment (Experiments):
Not clear how this extends to other approaches such as LLaVA. Results are shown only for Min-GPT4 extension.

Author response:
In light of your feedback, we **extended our approach to more advanced models such as LLaVA-v1.5**to offer a broader view of our approach's effectiveness. LLaVA-v1.5 has improved its image input resolution from 224 pixels to 336 pixels. It is trained on a diverse dataset comprising about 665K instruction-following data. Additionally, LLaVA-v1.5* is fine-tuned on SparklesDialogue (about 6.5K) using the low-resource technique LoRA to conserve memory and time. The results are shown below and have **updated in the revised manuscript in Section 5.1 and Table 2.**

| Model        | Instruction Data| BISON | NLVR2 | SparklesEval |
|--------------|-----------------|-------|-------|-------|
| GPT-4        | -               | -     | -     | 9.26  |
| MiniGPT-4    | description     | 46.0% | 51.3% | 3.91  |
| MiniGPT-4*   | description     | 51.3% | 46.7% | 3.50  |
| LLaVA-v1.5   | Mixture (665K ) | 52.7% | 53.3% | 2.75  |
| LLaVA-v1.5*  | **+SparklesDialogue**| 65.3% | 56.7% | 7.93  |
| SparklesChat | description     | 52.0% | 48.0% | 3.06  |
| SparklesChat | reasoning       | 52.7% | 54.0% | 6.71  |
| SparklesChat | SparklesDialogue| 56.7% | 58.0% | 8.56  |

According to the results, despite LLaVA-v1.5's advantages of higher image resolution (336 vs. 224 pixels) and a larger training set (665K vs. 6.5K), SparklesChat significantly outperforms LLaVA-v1.5 in three evaluation sets involving multiple images. While LLaVA-v1.5 outperforms MiniGPT-4 on BISON and NLVR2, it shows weaker results on SparklesEval. This may be due to LLaVA-v1.5's training data primarily focusing on closed-set multimodal tasks such as VQA, TextCaps, and RefCOCO, while lacking in open-ended dialogue training. After fine-tuning with our SparklesDialogue, LLaVA-v1.5* not only significantly improved in open-ended dialogue tasks but enhanced traditional multimodal tasks. These results validate the adaptability of our method in unlocking chats across multiple images for multimodal instruction-following models with minimal additional training cost.

Reviewer comment (Experiments):
Section 5.2 mentions, SparklesDialogueVG and SparklesEval use the same sources of images and captions. This is suspected to be one of the reasons why model trained on SparklesDialogueVG performs better than model trained on SparklesDialogueCC. Isn’t this a serious issue, especially since SparklesDialogueVG is claimed to be the high quality subset?

Author response:
Although there are similarities in image sources between SparklesDialogueVG and SparklesEval, the test set features unique combinations (e.g., 2-2 image pairings not present in training). The test set is carefully curated to ensure diversity, allowing for a comprehensive evaluation of multi-image dialogue capabilities. We acknowledge the similarity in image sources as a concern and plan to introduce additional data sources to provide a more balanced assessment.

Reviewer comment (Experiments):
Considering the current status of single-image comprehension, which still requires further advancements, it appears that addressing scenarios involving multiple images may not be an immediate priority. Additionally, when considering the data construction approach described in the paper, it becomes evident that the model's capabilities are still constrained by the limitations of single-image understanding.

Author response:
We acknowledge the importance of single-image understanding in visual comprehension. However, our focus on multi-image dialogues stems from real-world complexity, where individuals often encounter and discuss multiple images simultaneously.
    - Since the release of existing multimodal models such as MiniGPT-4 and LLaVA, there have been requests for multi-image capabilities in issue trackers that remain unresolved (see [issues 1](https://github.com/haotian-liu/LLaVA/issues/197), [2](https://github.com/Vision-CAIR/MiniGPT-4/issues/180), and [3](https://github.com/Vision-CAIR/MiniGPT-4/pull/232)). Our work aims to address these demands with our dataset and methodology. 
    - What's more, advancing multi-image dialogue capabilities complements single-image comprehension. Tackling complex scenarios with multiple images helps uncover limitations in single-image understanding and drives the field forward.

Reviewer comment (Novelty):
In my personal opinion, focusing on improving single-image comprehension would be more beneficial at this stage. Once single-image understanding is well-established, the demonstrated ability to handle multiple images, as showcased in the paper, should not pose significant challenges. It is crucial to ensure a solid foundation in single-image comprehension before delving into more complex scenarios involving multiple images.

Author response:
We acknowledge the importance of single-image understanding in visual comprehension. However, our focus on multi-image dialogues stems from real-world complexity, where individuals often encounter and discuss multiple images simultaneously.
    - Since the release of existing multimodal models such as MiniGPT-4 and LLaVA, there have been requests for multi-image capabilities in issue trackers that remain unresolved (see [issues 1](https://github.com/haotian-liu/LLaVA/issues/197), [2](https://github.com/Vision-CAIR/MiniGPT-4/issues/180), and [3](https://github.com/Vision-CAIR/MiniGPT-4/pull/232)). Our work aims to address these demands with our dataset and methodology. 
    - What's more, advancing multi-image dialogue capabilities complements single-image comprehension. Tackling complex scenarios with multiple images helps uncover limitations in single-image understanding and drives the field forward.

Reviewer comment (Experiments):
Considering the impressive performance of GPT-4 with ground truth (gt) annotation, could the authors provide a baseline using a strong caption model with an instruction-tuned Language Model to address the challenges raised in the paper?

Author response:
We appreciate your suggestion to consider a baseline using a strong caption model combined with an instruction-tuned language model. We analyze the advantages of multimodal models over this baseline lie in that (1) some images are challenging to describe solely through language, as visual elements can be more expressive than words, and (2) multimodal models can interpret and discuss visual elements in specific contexts. In light of your suggestion, we plan to experiment with this in future iterations to provide a more comprehensive comparison.

Reviewer comment (Experiments):
Does the model in the paper have the capability to handle scenarios with more than two images, considering that the paper only showcases examples with two images?

Author response:
SparklesChat is capable of handling dialogues involving multiple images, not limited to two. **We include examples with four or five images in Appendix F of the revised manuscript to demonstrate this expanded capability.**
    - To analyze SparklesChat's ability to generalize in scenarios involving a larger number of images, we consolidated several questions from a dialogue into a single query. This was done to generate responses for queries involving four or five images, as shown in **Figure 9 and Figure 10** in the updated manuscript, where the model effectively distinguishes between the images and adheres to complex queries. In the case of Figure 10, where three questions involving five images are concatenated into one query, the model tends to ignore the final question and only responds to the first two. We believe this limitation arises from the absence of similar patterns in the training data. A potential solution could involve incorporating multiple turns into each training dialogue to enhance the model's ability to handle such complex scenarios.

Reviewer comment (Experiments):
The simple data construction approach fails to yield effective samples. In Figure 3, the response from GPT-4 seems too naive, *i.e.*, in image #1, we see ..., in image #2, we witness..... This is just a concatenation of descriptions of two images.

Author response:
Our approach to generating visual dialogue with multiple images extends beyond simple image-text pair linking or descriptions. We instruct GPT-4 to simulate realistic user-assistant interactions as introduced in Section 4.1, and not to directly describe images with the prompt *"Please make sure not to reveal the content of the images or describe the images in the user messages"* as shown in Table 8. Thus, the generated dialogues are not mere concatenations of descriptions but are contextually rich and coherent conversations. 
   - Sometimes, a detailed image description is part of the image understanding and reasoning process. The case in Figure 3 is picked to show a response that clearly distinguishes between multiple images. Meanwhile, Figure 19 (original Figure 17) presents a more complex example of generating a promotional text inspired by several images but not describing them naively.
   - The statistics in Figure 4 further illustrate the diversity of SparklesDialogue, showcasing its range from generating text materials to seeking advice or discussing relationships between images. This involves reasoning about the content and context of the images, connecting different visual elements, and generating responses that reflect a deep understanding of the images about the user's queries.

Reviewer comment (Experiments):
Current experiments are insufficient and cannot form a strong foundation to support the effectiveness of your model and training data.

Author response:
Thanks for your constructive suggestion! 

    **Expanded Baselines.** SparklesChat is built upon the MiniGPT-4 and adapted to accommodate multiple images due to its relevance and prominence in the field. We **extended our approach to more advanced models such as LLaVA-v1.5** to offer a broader view of our approach's effectiveness. LLaVA-v1.5 has improved its image input resolution from 224 pixels to 336 pixels. It is trained on a diverse dataset comprising about 665K instruction-following data. Additionally, LLaVA-v1.5* is fine-tuned on SparklesDialogue (about 6.5K) using the low-resource technique LoRA to conserve memory and time. The results are shown below and have **updated in the revised manuscript in Section 5.1 and Table 2.**

    | Model        | Instruction Data| BISON | NLVR2 | SparklesEval |
    |--------------|-----------------|-------|-------|-------|
    | GPT-4        | -               | -     | -     | 9.26  |
    | MiniGPT-4    | description     | 46.0% | 51.3% | 3.91  |
    | MiniGPT-4*   | description     | 51.3% | 46.7% | 3.50  |
    | LLaVA-v1.5   | Mixture (665K ) | 52.7% | 53.3% | 2.75  |
    | LLaVA-v1.5*  | **+SparklesDialogue**| 65.3% | 56.7% | 7.93  |
    | SparklesChat | description     | 52.0% | 48.0% | 3.06  |
    | SparklesChat | reasoning       | 52.7% | 54.0% | 6.71  |
    | SparklesChat | SparklesDialogue| 56.7% | 58.0% | 8.56  |

    According to the results, despite LLaVA-v1.5's advantages of higher image resolution (336 vs. 224 pixels) and a larger training set (665K vs. 6.5K), SparklesChat significantly outperforms LLaVA-v1.5 in three evaluation sets involving multiple images. While LLaVA-v1.5 outperforms MiniGPT-4 on BISON and NLVR2, it shows weaker results on SparklesEval. This may be due to LLaVA-v1.5's training data primarily focusing on closed-set multimodal tasks such as VQA, TextCaps, and RefCOCO, while lacking in open-ended dialogue training. After fine-tuning with our SparklesDialogue, LLaVA-v1.5* not only significantly improved in open-ended dialogue tasks but enhanced traditional multimodal tasks. These results validate the adaptability of our method in unlocking chats across multiple images for multimodal instruction-following models with minimal additional training cost.

Reviewer comment (Experiments):
Regarding baselines, you compare your method only with MiniGPT-4, which in the reviewer's understanding is an embarrassingly weak and simple model and dataset. More comparisons are definitely needed.

Author response:
Thanks for your constructive suggestion! 

    **Expanded Baselines.** SparklesChat is built upon the MiniGPT-4 and adapted to accommodate multiple images due to its relevance and prominence in the field. We **extended our approach to more advanced models such as LLaVA-v1.5** to offer a broader view of our approach's effectiveness. LLaVA-v1.5 has improved its image input resolution from 224 pixels to 336 pixels. It is trained on a diverse dataset comprising about 665K instruction-following data. Additionally, LLaVA-v1.5* is fine-tuned on SparklesDialogue (about 6.5K) using the low-resource technique LoRA to conserve memory and time. The results are shown below and have **updated in the revised manuscript in Section 5.1 and Table 2.**

    | Model        | Instruction Data| BISON | NLVR2 | SparklesEval |
    |--------------|-----------------|-------|-------|-------|
    | GPT-4        | -               | -     | -     | 9.26  |
    | MiniGPT-4    | description     | 46.0% | 51.3% | 3.91  |
    | MiniGPT-4*   | description     | 51.3% | 46.7% | 3.50  |
    | LLaVA-v1.5   | Mixture (665K ) | 52.7% | 53.3% | 2.75  |
    | LLaVA-v1.5*  | **+SparklesDialogue**| 65.3% | 56.7% | 7.93  |
    | SparklesChat | description     | 52.0% | 48.0% | 3.06  |
    | SparklesChat | reasoning       | 52.7% | 54.0% | 6.71  |
    | SparklesChat | SparklesDialogue| 56.7% | 58.0% | 8.56  |

    According to the results, despite LLaVA-v1.5's advantages of higher image resolution (336 vs. 224 pixels) and a larger training set (665K vs. 6.5K), SparklesChat significantly outperforms LLaVA-v1.5 in three evaluation sets involving multiple images. While LLaVA-v1.5 outperforms MiniGPT-4 on BISON and NLVR2, it shows weaker results on SparklesEval. This may be due to LLaVA-v1.5's training data primarily focusing on closed-set multimodal tasks such as VQA, TextCaps, and RefCOCO, while lacking in open-ended dialogue training. After fine-tuning with our SparklesDialogue, LLaVA-v1.5* not only significantly improved in open-ended dialogue tasks but enhanced traditional multimodal tasks. These results validate the adaptability of our method in unlocking chats across multiple images for multimodal instruction-following models with minimal additional training cost.

Reviewer comment (Evaluation):
Regarding evaluation benchmarks, you use three benchmarks for evaluation — BISON, NLVR2, and your own evaluation data. Among them, BISON and NLVR2 are not commonly used benchmarks now. Furthermore, on your own evaluation data, you claim your performance approaches GPT-4. However, since your self-constructed training data could share a similar distribution to your evaluation data, the reviewer thinks this claim is not well established.

Author response:
**Evaluation Benchmarks.** Our SparklesEval benchmark is designed to evaluate open-ended dialogues involving multiple images and turns, using LLMs as judges. Additionally, NLVR and BISON were specifically chosen for their use of natural images and the feasibility of automatic evaluation with unambiguous answers, such as TRUE/FALSE or single-choice problems. In response to your suggestion, we are expanding our experimental framework to include broader evaluation benchmarks, incorporating images from diverse domains (e.g., documents, fashion, cartoons) and a wider range of evaluation metrics beyond accuracy measures. Additionally, we plan to diversify SparklesEval with different distributions of images and text to reduce potential biases.