Paper: Delving into LLMs’ visual understanding ability using SVG to bridge image and text (ICLR-2024)

Reviewer comment (Experiments):
For the evaluation of Multimodal LLMs, the authors only experiment on LLaVa and conclude that "This behavior underscores the limitations of current large multimodal models in structured and sophisticated reasoning.", which is not rigorous. The authors should evaluate more Multimodal LLMs such as BLIP-2, InstructBLIP, MiniGPT-4, mPLUG-Owl, etc.

Author response:
We appreciate your suggestion. We have expanded our comparison in the table to include additional Multimodal LLMs. The updated results indicate that combining SVG with LLMs outperforms other models like BLIP-2, InstructBLIP, MiniGPT-4, and mPLUG-Owl in handling complex, structured reasoning tasks.

| Question Type | GPT-CoT | LLaVa | CNN+MLP | Relation Networks | InstructBLIP (13b) | BLIP2 (Flan T5-xxl) | mPLUG_owl | MiniGPT4 (13B) |
|---------------|---------|-------|---------|-------------------|--------------------|---------------------|-----------|----------------|
| Format        | SVG     | PNG   | PNG     | PNG               | PNG                | PNG                 | PNG       | PNG            |
| Unary         | 0.90    | 0.60  | 0.65    | 0.89              | 0.53               | 0.50                | 0.38      | 0.53           |
| Binary        | 0.95    | 0.60  | 0.75    | 0.80              | 0.53               | 0.53                | 0.63      | 0.55           |
| Ternary       | 0.88    | 0.10  | 0.55    | 0.55              | 0.10               | 0.30                | 0.30      | 0.30           |
| Average       | 0.89    | 0.43  | 0.65    | 0.75              | 0.38               | 0.44                | 0.43      | 0.46           |

_Caption: Performance comparison across different models on the Sort-of-Clever Dataset._

Reviewer comment (Experiments):
The data for the evaluation of both the discriminative and generative visual understanding tasks are relatively simple. How does the LLM perform with SVG representations on more complicated images? Since the authors also mention that SVG is not effective in handing photographic content, using SVG with LLMs to tackle visual tasks may not be effective enough.

Author response:
Thank you for raising this concern. We agree that the current approach of integrating SVG with LLMs for complex visual tasks may fall short, primarily due to the loss of fine details in converting and the lengthy sequences of encoding extensive details. To tackle this, a potential solution is to leverage large vision models to extract the essence of images and convert that essence into the SVG format for LLM processing. An example is utilizing SAM [1] for object mask segmentation, which is then translated to SVG for LLMs, as depicted in Figure 6 (d). We then show a recognition result in Figure 7 (b).

[1] Kirillov, Alexander, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao et al. "Segment anything." ICCV (2023).

Reviewer comment (Experiments):
The SVG representation may not capture all the nuances of an image. It is yet unclear how well this method would generalize to more complex images or visual tasks.

Author response:
Thank you for pointing out this concern. We acknowledge the limitations of using SVG with LLMs for complex visual tasks, due to the loss of fine-grained details during conversion and the challenge of managing prohibitively long sequences when encoding detailed photographic content. To tackle this, a potential solution is to leverage large vision models to extract the essence of images and convert that essence into the SVG format for LLM processing. An example is utilizing SAM [1] for object contour segmentation, which is then translated to SVG for LLMs, as depicted in Figure 6 (d). We then show a recognition result in Figure 7 (b).

Reviewer comment (Experiments):
The issue raised in the introduction about whether LLMs can learn world models without grounding in physical interaction and visual perception remains unaddressed. A comparison of LLMs' performance using SVGs with traditional vision-based models could have provided more context about the relative effectiveness of this approach.

Author response:
Thank you for your suggestion. We'd like to clarify that in our paper, we have compared the performance of LLMs using SVGs with traditional vision-based models. This includes:

- Visual reasoning tasks in Table 1, where LLMs with SVGs outperform SoTA vision-based models in both in-distribution and out-of-distribution scenarios.

- Image classification tasks in Table 2, showing LLMs with SVGs again surpassing SoTA vision-based models under similar conditions.

- Visual prompting in Table 3, where LLMs with SVGs demonstrate superior performance compared to SoTA vision-based models.

Reviewer comment (Evaluation):
How well do SVG representations capture complex images? Could you provide instances or examples where this approach has limitations?

Author response:
Thank you for pointing out this concern. We acknowledge the limitations of using SVG with LLMs for complex visual tasks, due to the loss of fine-grained details during conversion and the challenge of managing prohibitively long sequences when encoding detailed photographic content. To tackle this, a potential solution is to leverage large vision models to extract the essence of images and convert that essence into the SVG format for LLM processing. An example is utilizing SAM [1] for object contour segmentation, which is then translated to SVG for LLMs, as depicted in Figure 6 (d). We then show a recognition result in Figure 7 (b).

Reviewer comment (Experiments):
What could be the potential impact if a minor percentage of errors into the representation of SVGs is introduced?

Author response:
Thanks for both of your suggestions. To rigorously evaluate the robustness of LLMs against variations in SVG data, we conducted three distinct experiments, where we: (i) shuffle the order of paths, (ii) randomize path coordinate replacement, and (iii) randomize string replacement. Each experiment was designed to mimic real scenarios of imperfections in SVG data, providing insights into using SVG with LLMs under challenging conditions.

__(i) Path Shuffle Experiment__ SVG data consists of multiple path elements, each representing an object or line in the image. In this experiment, we tested the model's ability to interpret hand-drawn SVG data from the MNIST dataset when the sequence of path elements was shuffled. Note that, by the very nature of SVG data, this alteration will __not__ change the final image; and hence, it is important to test if the LLM can remain invariant to this change, even though we have not explicitly provided it with this knowledge. 
The goal is to assess the model's ability to correctly interpret the digit, regardless of the order of path elements. The results are summarized in the table below. Results indicate that SVG is robust to path shuffle at test time, maintaining a comparable performance compared to the SVG representation with the original path ordering.


| Condition       | Accuracy (%) |
|-----------------|--------------|
| Without Shuffle | 99.10        |
| With Shuffle    | 98.74        |

_Caption: Model accuracy for the path shuffle experiment._

__(ii) Random Path Coordinate Replacement__
We follow the reviewer's suggestion and introduce a subtle form of noise by randomly altering the coordinates in the SVG path data. Each numerical value within the path commands is randomly translated within a specific range $k$. For example, for $k=1$,   `... <path d='M0 0 C18 0 17...'>` might become `... <path d='M1 0 C18 1 18...'>`. We test several variations: (i) a minor adjustment within a 1/28th range (reflecting the 28x28 resolution of MNIST) and (ii) a more significant alteration within a 5/28th range. This simulates real-world scenarios of minor inaccuracies in SVG data, such as those resulting from conversion errors or imprecise digitization.
Table shown below presents the accuracy under different noise scales. Results indicate data SVG representation is decently robust to the perturbation of the coordinate values. 

| Noise Scale | Accuracy (%) |
|-------------|--------------|
| 0/28        | 99.10        |
| 1/28        | 98.97        |
| 2/28        | 97.91        |
| 5/28        | 87.56        |

_Caption: Model accuracy under different levels of coordinate noise._

__(iii) Random String Replacement__
Finally, we design the most aggressive test of robustness, where we replace random characters in the SVG strings with any English alphabet letter, digit, or special symbol, regardless of whether they are numerical values or part of SVG commands (like `transform='translate(0,0)'`). The experiment is conducted with varying probabilities for character replacement, as shown below.
Surprisingly, even after replacing 20\% of a string with random characters, SVG with LLMs maintains a high accuracy rate of 90.79\%. This suggests that SVG with LLMs can handle a wide range of perturbations in SVG data.


The following table presents the accuracy with varying probabilities of random string replacement:

| Replacement Probability (%) | Accuracy (%) |
|-----------------------------|--------------|
| 0                           | 99.10        |
| 1                           | 99.06        |
| 5                           | 98.40        |
| 10                          | 97.40        |
| 20                          | 90.79        |
| 50                          | 39.38        |

_Caption: Accuracy with different probabilities of random string replacement._

In conclusion, the results from these experiments demonstrate the robustness of using SVG data with LLMs for visual understanding. Despite the introduction of perturbations, using SVG data with LLMs can perform well under challenging conditions.

Reviewer comment (Novelty):
**Component Originality:** While the experiments are detailed, the novelty of the individual components used within the work is unclear. The application of Scalable Vector Graphics (SVG), Large Language Models (LLMs), the dataset selection, the tasks, and the chosen evaluation metrics all appear to be repurposed from existing literature without significant innovation or new application. For a stronger contribution, the authors could benefit from integrating at least one novel element or a unique combination of these elements that distinguishes this work from prior studies.

Author response:
> “While the experiments are detailed, the novelty of the individual components used within the work is unclear”


We wish to clarify that our study primarily aims to provide a thorough analysis of the strengths and weaknesses of employing LLMs to understand and reason about images without visual data. Below, we detail the novelty of each component in our study.


- Comprehensive study of LLMs across various visual tasks using SVG format, including visual reasoning, image classification, and image generation.
- Investigation into LLMs' performance in visual data using SVG format under conditions of distribution shifts, expanding our understanding of their robustness.
- Analysis of LLMs' generative capabilities in image generation and editing based on interactive feedback within the SVG format.

Reviewer comment (Experiments):
**Source of Performance Discrepancy:** The comparative results between LLM+SVG and CNN+PNG in the first experiment are intriguing but lack a clear explanation. The performance gap could be attributed to differences in input representation or model architecture, among other factors. An in-depth ablation study could help isolate the impact of each component. Furthermore, the disparity in model sizes (e.g., GPT-4's size relative to that of a CNN) is a confounding variable that merits consideration. To convincingly argue the advantages of using SVG for image representation in LLMs, the authors should compare SVG with alternative representations using the same underlying model architecture, such as GPT-4.

Author response:
> “The comparative results between LLM+SVG and CNN+PNG in the first experiment are intriguing but lack a clear explanation. ... isolate the impact of each component.”

Thank you for your suggestion. Following your advice, we conducted additional experiments to assess the impact of different input formats using the same model, as shown in the table below. Specifically, we compare the SVG input format with GPT4 against PNG input format with GPT4V (likely larger than GPT4 due to its visual encoding network component) for a balanced evaluation. The results indicate that using SVG with GPT4 significantly outperforms using PNG with GPT4V, in complex, structured reasoning tasks.



| Question Type | GPT-CoT | GPT-4V | LLaVa | CNN+MLP | Relation Networks | InstructBLIP (13b) | BLIP2 (Flan T5-xxl) | mPLUG_owl | MiniGPT4 (13B) |
|---------------|---------|--------|-------|---------|-------------------|--------------------|---------------------|-----------|----------------|
| Format        | SVG     | PNG    | PNG   | PNG     | PNG               | PNG                | PNG                 | PNG       | PNG            |
| Unary         | 0.90    | 0.75   | 0.60  | 0.65    | 0.89              | 0.53               | 0.50                | 0.38      | 0.53           |
| Binary        | 0.95    | 0.74   | 0.60  | 0.75    | 0.80              | 0.53               | 0.53                | 0.63      | 0.55           |
| Ternary       | 0.88    | 0.28   | 0.10  | 0.55    | 0.55              | 0.10               | 0.30                | 0.30      | 0.30           |
| Average       | 0.89    | 0.59   | 0.43  | 0.65    | 0.75              | 0.38               | 0.44                | 0.43      | 0.46           |

_Caption: Performance comparison across different models on the Sort-of-Clever Dataset._





Additionally, to ensure a fair comparison, we compared the performance of both SVG and PNG inputs using the same Vicuna model for image classification in the table below 
As shown in the following figure, under the same model architecture, results indicate that SVG is more robust than rasterized representation. 





| Method                 | ConvNeXt (fine-tuning) | Vicuna (fine-tuning) | Vicuna (fine-tuning) |
|------------------------|------------------------|----------------------|----------------------|
| Image Format           | PNG                    | SVG                  | PNG                  |
| MNIST                  | 99.5%                  | 99.1%                | 99.4%                |
| CMNIST-(A)             | 79.5%                  | 95.7%                | 42.9%                |
| CMNIST-(B)             | 32.6%                  | 92.9%                | 24.8%                |

_Caption: Performance comparison across different models on the MNIST Dataset._

Reviewer comment (Experiments):
**Dataset and Task Relevance:** The chosen dataset and task do not seem to reflect complex real-world scenarios and may inherently favor SVG representations due to their structured nature. This could introduce a bias towards the advantages of SVG in image understanding tasks that focus on structural rather than fine-grained details. The authors would enhance the robustness of their findings by including a broader range of tasks with varying degrees of complexity and realism to demonstrate the efficacy of SVG representations across different contexts.

Author response:
> “The chosen dataset and task do not seem to reflect complex real-world scenarios and may inherently favor SVG representations due to their structured nature.”



We thank the reviewer for raising and highlighting an important point. First, we would like to re-emphasize the main goal of our work: to see how much knowledge LLMs already have in understanding visual data, even though one might not naturally assume them to be suited for any vision task. Therefore, any experiments that we conduct (e.g., Table 1) should be seen as a way to study that question, and _not necessarily_ as a way to demonstrate that LLMs are fundamentally superior to CNN/transformers. And since, to the best of our knowledge, our work is the first to thoroughly study this hidden capability of LLMs, we have chosen the simpler settings. Our hope, however, is that our work can motivate investigations into other, more complex, vision tasks that can be solved with the help of an LLM.

Reviewer comment (Novelty):
**Methodological Limitations:** The exploration of image understanding through LLMs is indeed crucial; however, the analysis presented in this work is constrained by its reliance on existing methodologies. The absence of innovative input representations, model adaptations, or new evaluation benchmarks suggests a missed opportunity for advancing the field. To forge a more impactful contribution, the authors should strive to develop and introduce novel methodologies or significantly adapt existing ones to the specific challenges of image understanding through LLMs.

Author response:
> “The analysis presented in this work is constrained by its reliance on existing methodologies. The absence of innovative input representations, model adaptations, or new evaluation benchmarks suggests a missed opportunity for advancing the field.”


We recognize that while our study relies on established techniques, these techniques are sufficiently effective for practical use in simple scenarios. We agree that addressing complex real-world images may necessitate the development of new methods. We are grateful for being the pioneers to start the investigation of LLMs for understanding and reasoning about images without relying on visual data, and we are committed to advancing our research to tackle real-world challenges more effectively.