Paper: Chain of Images for Intuitively Reasoning (ICLR-2024)

Reviewer comment (Novelty):
The novelty is relatively limited. There are some similar methodologies for multi-modal large language models to generate images (related work Sec.4.2). The paper is more concerned about using multi-modality to assist language reasoning. The claimed novelty is not that strong -- it is a less-studied task, rather than a new methodology.

Author response:
- Unlike MLLMs that focus primarily on solving VQA tasks with a given ground truth image, our Chain of Images (CoI) method tackles more complex tasks by firstly generating images from textual queries. This approach goes beyond merely aligning image features with textual contexts, as is common in VQA tasks. CoI directly incorporates images into the reasoning process, representing a more advanced multimodal reasoning form, thereby enhancing the model's capabilities beyond text-only analysis.
    - The SyMLLM framework is different from other MLLMs like NExT-GPT, DreamLLM, and MiniGPT-5, which use diffusion models for pixel image generation. SyMLLM excels in creating vector images, offering enhanced control and suitability for complex reasoning tasks, thereby providing a novel approach for multimodal generation models.

Reviewer comment (Experiments):
The setting of commonsense tasks is confusing. It seems that the image is generated by stable diffusion instead of the proposed SyMLM. Thus, it seems not a good evaluation for the proposed method.

Author response:
- CoI primarily focuses on image-centered reasoning and is not solely dependent on SyMLLM. We have conducted experiments with GPT-4V and LLaVA across four tasks, and explored the combination of Stable Diffusion XL and DALL·E 3 for image generation with LLaVA/GPT-4V for image recognition. This approach allows us to evaluate CoI's effectiveness in commonsense datasets.

Reviewer comment (Experiments):
To generate the Chain of Images, extra training is needed for each task. This largely hurts the universal capability of large language models and hinders broader application. Also, given the diverse reasoning tasks large language models can solve, the experiments cannot show the effectiveness of the method on other reasoning tasks.

Author response:
- Our experiments, as shown in Table 3, demonstrate that in-context learning can effectively generate complex images without extensive training. This proves that our method can maintain the universal capability of large language models and does not require significant additional training for each task.

Reviewer comment (Writing):
In Section 2.3 ("Converting Symbols To Image"), the exposition on SVG and FEN formats is overly succinct. It is unclear to the readers how these formats facilitate the effective translation of symbols into images.

Author response:
In all datasets, we uniformly utilize the SVG format to boost the method's universality. SVG files can be directly displayed on screens or easily converted into formats like PNG or JPEG using tools like CairoSVG.

Reviewer comment (Reproducibility):
Regarding Figure 5, the manuscript does not elucidate how the question text "a line segment from (0.5, 3.4) to (3.5, -4.2)" translates into SVG format. It is unclear if there is a specific prompt used for this translation.

Author response:
The SVG format follows a fixed syntax, which can be generated using prompts. The coordinates for SVG elements like cycles, lines, and polygons are readily extracted from the question text.

Reviewer comment (Presentation):
Regarding Figure 5, the figure directly presents intersection points as intermediate answers without explaining the rationale or method behind their determination.

Author response:
Once a question is converted into an image, it becomes easy for both human observers and our SyMLLM to identify elements like intersection points at a glance.

Reviewer comment (Presentation):
Table 1 suffers from a lack of detailed analysis, particularly in that the term "geometric shapes in each sample, representing the different difficulty levels" is ambiguous. Providing illustrative examples would be beneficial.

Author response:
Thank you for your suggestion. In the revised paper, we have thoroughly addressed the three issues you raised, providing a more detailed analysis in the revised paper.

Reviewer comment (Reproducibility):
In Section 2.2, the paper states, "Subsequently, these images are converted into image embeddings by the image encoder. The embeddings are then concatenated with the text embeddings to generate the next token." It is unclear whether the image embeddings can be directly concatenated with the text embeddings without employing any adapter layers or fine-tuning mechanisms.

Author response:
SyMLLM's architecture, similar to LLaVA, includes an image encoder, an LLM, and a projection layer. SyMLLM is distinct in its ability to generate precise images from textual instructions, thanks to fine-tuning on text-to-symbolic datasets.

Reviewer comment (Evaluation):
Also in Section 2.2, the claim is made that "In subsequent experiments, we observe that the accuracy of image generation approaches nearly 100%." The metrics used to quantify the similarity between the generated and target images are not explained.

Author response:
In the revised version of the paper, we have redefined the metrics used. Specifically, for the geometric dataset, accuracy is calculated as the number of correctly predicted shapes per data point divided by the total number of shapes. For the chess dataset, accuracy is determined by the number of correctly predicted chess pieces divided by the total number of positions.

Reviewer comment (Reproducibility):
In Section 2.3, the text mentions, "The image is then used to count intersection points." The methodology employed for counting the intersection points within an image in the context of this work is not specified.

Author response:
For image encoding, identifying intersection points in an image requires basic pattern recognition capabilities. An intersection point is identified by the crossing of two different colors. This method is efficiently integrated into our model's processing capabilities.

Reviewer comment (Experiments):
The tested domains are domains in which image generation should intuitively help. However, it is plausible that there are domains where incorporating image is not significantly helpful (e.g. arithmetics). It would be better if the authors have tried their method in more diverse domains.

Author response:
Regarding arithmetic, it is true that LLMs can potentially solve these problems in text space. Our experiments, however, focus on domains like Geometric, Chess, Topological, and Temporal, demonstrating the potential for more diverse applications of our method in more complex tasks.

Reviewer comment (Experiments):
In domains other than geometry, the LLM is generating only a single image. The authors should try more domains in which chain of image generation is needed. Moreover, in the geometric domain, the authors could compare their method against a simple baseline which generates the final image at once without chaining process.

Author response:
We have compared the use of single v.s. multiple images in Tables 1 and 2. The results clearly indicate that using multiple images, especially in increasingly complex tasks, offers significant benefits over a single image.

Reviewer comment (Experiments):
As the proposed method is proposing to also use image along with text, the authors should have tried more extensive experiments using VLM using / not using their method.

Author response:
VLMs typically solve VQA tasks which necessitate a ground truth image for answering questions about the image. Our CoI method, in contrast, is designed for more complex tasks where no ground truth image exists. We also explored the integration of VLMs with our text-to-symbolic prompts, converting them to a SyMLLM without finetuning. This is detailed in Tables 1 and 2, where LLaVA was able to generate relevant images, but struggled with abstract imagery.

Reviewer comment (Experiments):
Rather than fine-tuning LLM to generate images given a set of problems and corresponding images to be given as a training data, the authors can consider reducing the size of training data and simply try in-context learning.

Author response:
We have conducted in-context learning experiments, as shown in Table 3, highlighting the potential effectiveness of our method with reduced training data. And observing that GPT-4V can generate images using 3-shot text-to-symbolic prompts, although they struggle with visual questions, leading to unsatisfactory outcomes.

Reviewer comment (Experiments):
Most of the results are either based on the perfect image generation ability of LLM after fine-tuned, or ground truth image.

Author response:
In Table 3, we also demonstrate the accuracy of images generated through in-context learning, showcasing that satisfactory results can be achieved without relying solely on perfect image generation or ground truth images.

Reviewer comment (Writing):
Although the authors included examples of common sense domain, it is not easy to understand the two tasks tested in common sense reasoning. For instance, the first task of determining the scene described in the text can be misinterpreted as a task in which VLM determines the best image given a set of single text description and multiple images.

Author response:
We have revised the section on the common sense dataset for clarity. The Location task is evaluated based on multiple-choice answers, where the inclusion of standard answers determines correctness. The Unusual task involves open-ended questions about common sense violations. Here, we adopted different metrics: if a large-scale model fails to recognize nonsensical elements in the description, the answer is deemed incorrect; otherwise, it is considered correct.

Reviewer comment (Experiments):
The authors didn't clearly state the results of a second task in common sense domain. The paper only contains two examples in Fig. 6 and Fig. 7.

Author response:
We have revised the section on the common sense dataset for clarity. The Location task is evaluated based on multiple-choice answers, where the inclusion of standard answers determines correctness. The Unusual task involves open-ended questions about common sense violations. Here, we adopted different metrics: if a large-scale model fails to recognize nonsensical elements in the description, the answer is deemed incorrect; otherwise, it is considered correct.

Reviewer comment (Experiments):
Lack of experiments comparing the diffusion-based model generated images vs the proposed more controllable image generation strategy. For example, a qualitative comparison can be shown in Figure 4 to show how the proposed method solve the issues presented. Besides, in Table 1/2, the ablations should also be shown to support the claim.

Author response:
1. **Experiment Comparison Between Diffusion-Based Models and Our Controllable Image Generation Strategy:**
    - We have compared the performance of NExT-GPT, a diffusion-based model, in Geometric and Chess tasks, as illustrated in Tables 1 and 2. The generated images and their respective analyses are detailed in Sections 4.1 and 4.2 of the revised paper. We assert that diffusion-based models are less effective in aiding abstract problem-solving.

Reviewer comment (Experiments):
In contribution, it says "counting the intersection points of geometric shapes, images provide an intuitive representation of the relationships (such as spatial, topological, temporal, etc.) between the items", but seems like only the spatial relationship is the primary point that's been validated in the the experiment 3.1.

Author response:
2. **Validation of Different Types of Relationships in Experiments:**
    - We have conducted experiments in Geometric, Chess, Topological, and Temporal tasks. The results, presented in Table 3, demonstrate CoI's efficacy in these areas, substantiating its capability to intuitively represent various relationships (spatial, topological, temporal, etc.) in problem-solving contexts. In the main paper, we have analyzed several failure cases and observed that GPT-4V demonstrates accurate capabilities in generating SVG images. However, the visual capabilities of GPT-4V are not as satisfactory, even in some simple examples. We believe that with the enhancement of visual processing abilities in large-scale models, there will be a significant improvement in this comparative outcome.

Reviewer comment (Reproducibility):
The details of the whole multimodal language model is lacked. Especially in Section 2.2, there supposed to be some vision encoder details and the training details. Although in Section3.2, it mentions clip-vit-large-patch14, the whole model details are missing.

Author response:
4. **Details of the Multimodal Language Model and Data Generation:**
    - The detailed methodologies for Data Processing, Model Structure, and Training for each task are described in Section 4 of the revised paper.

Reviewer comment (Experiments):
How does this method compare with LLaVA? Is this method also benefiting multimodal large language models? or only get some advantage when compared with LLMs? Then is that really a fair comparison?

Author response:
3. **Comparison with LLaVA-Based Models and Other Multimodal LLMs:**
    - SyMLLM's architecture shares similarities with LLaVA, including an image encoder, an LLM, and a projection layer. However, unlike LLaVA models which are limited to image interpretation, SyMLLM is fine-tuned on text-to-symbolic datasets, enabling precise image generation from textual instructions.
    - For LLaVA-like MLLMs, their primary function is solving VQA tasks requiring a ground truth image. In contrast, CoI tackles more complex tasks by firstly generating images from textual queries. We observed that LLaVA-like MLLMs, when combined with our text-to-symbolic prompts, could adapt to SyMLLM without fine-tuning. This finding, detailed in Tables 1 and 2, shows that LLaVA can generate relevant images, surpassing diffusion-based models in this aspect.
    - We have also compared our method with NExT-GPT, capable of image generation using stable diffusion 1.5. The results, presented in Tables 1 and 2, indicate it's failed in both image generation and task-solving capabilities compared to CoI.