Paper: ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models (ICLR-2024)

Reviewer comment (Writing):
There is no explicit definition of VCI.

Author response:
We would like to kindly point out that the definition of VCI is in the Sec. 3.2 of the paper. We’ve made the distinction between VCI and VCU clearer in our revision.

Reviewer comment (Novelty):
In fact, the definition of VCU is no different from that of image text retrieval.

Author response:
We appreciate the reviewer's insight regarding the connection between VCU problems and image-text retrieval. While acknowledging this connection, we argue that this relationship is not a limitation of our work. As per the ICLR 2024 Reviewer instructions, “originality may arise from […] **creative combinations of existing ideas, application to a new domain**.” In this vein, our contribution is unique in that we are the first one to solve a subset of VCR, namely VCU, using image-text alignment capabilities in VLMs. We believe that our approach of categorizing problems is significant, as it has the potential to reduce costly API requests to LLMs.

Reviewer comment (Presentation):
Fig.4, I think, should be changed with a table rather than drawing a table.

Author response:
We appreciate the reviewer for the suggestion. However, we believe combining the example images and the analysis as a figure does not affect the presentation.

Reviewer comment (Experiments):
Judging from Table1, the number of examples studied is very small. It is unclear if the evidence derived from the results is robust.

Author response:
2. **Results on more examples and decoding configurations of VCR dataset.** Thank you for the suggestion. We agree running on more examples and more decoding configurations could better help validate our findings. Therefore, we sample **3000 examples** from the VCR dataset. The results are shown below: 
| Model           | VCU conf | VCU !conf | VCI conf | VCI !conf |
|-----------------|----------|-----------|----------|-----------|
| BLIP2-Pretrain  | 70.0     | 56.3      | 59.2     | 47.4      |
| BLIP2-Pretrain-clue | 70.6 | 56.7      | 63.3     | 49.2      |
| Caption             | 75.3     | 46.6          | 65.3     | 41.9      |
| Caption + VQA Clue | 75.9  | 51.9      | 65.3     | 47.3      |
| Caption + LLM Clue | 72.9  | 58.1      | 57.1     | 52.9      |

    As shown above, the conclusions drawn from Table 1, and the branching strategy explained above (and in the revision of the paper) remain unchanged compared with the results on 500 examples.

    We also run the **[Caption + LLM Clue]** version in Table 1 with extra temperatures and with different in-context examples in the prompt for thinking visual observations to support each choice (Fig.3 Right).

    | Caption + LLM Clue  | VCU conf | VCU !conf | VCI conf | VCI !conf |
    |-----------------|----------|-----------|----------|-----------|
    | Original        | 72.9     | 58.1      | 57.1     | 52.9      |
    | temp 0.1        | 72.4     | 58.8      | 57.1     | 53.9      |
    | Temp 0.2        | 72.4     | 58.5      | 61.2     | 52.2      |
    | ICL example     | 74.7     | 56.7      | 59.2     | 54.2      |

   We can also observe that, using different LLM temperature and in-context examples only have a small influence on the performance, and does not affect the conclusions. We’ve updated the results in the revision.

Reviewer comment (Experiments):
No multiple runs across decoding configs (e.g. temperature, selection of in-context examples). This limits the robustness and generality of the findings.

Author response:
2. **Results on more examples and decoding configurations of VCR dataset.** Thank you for the suggestion. We agree running on more examples and more decoding configurations could better help validate our findings. Therefore, we sample **3000 examples** from the VCR dataset. The results are shown below: 
| Model           | VCU conf | VCU !conf | VCI conf | VCI !conf |
|-----------------|----------|-----------|----------|-----------|
| BLIP2-Pretrain  | 70.0     | 56.3      | 59.2     | 47.4      |
| BLIP2-Pretrain-clue | 70.6 | 56.7      | 63.3     | 49.2      |
| Caption             | 75.3     | 46.6          | 65.3     | 41.9      |
| Caption + VQA Clue | 75.9  | 51.9      | 65.3     | 47.3      |
| Caption + LLM Clue | 72.9  | 58.1      | 57.1     | 52.9      |

    As shown above, the conclusions drawn from Table 1, and the branching strategy explained above (and in the revision of the paper) remain unchanged compared with the results on 500 examples.

    We also run the **[Caption + LLM Clue]** version in Table 1 with extra temperatures and with different in-context examples in the prompt for thinking visual observations to support each choice (Fig.3 Right).

    | Caption + LLM Clue  | VCU conf | VCU !conf | VCI conf | VCI !conf |
    |-----------------|----------|-----------|----------|-----------|
    | Original        | 72.9     | 58.1      | 57.1     | 52.9      |
    | temp 0.1        | 72.4     | 58.8      | 57.1     | 53.9      |
    | Temp 0.2        | 72.4     | 58.5      | 61.2     | 52.2      |
    | ICL example     | 74.7     | 56.7      | 59.2     | 54.2      |

   We can also observe that, using different LLM temperature and in-context examples only have a small influence on the performance, and does not affect the conclusions. We’ve updated the results in the revision.

Reviewer comment (Evaluation):
Judging from Table1's LLM section: I'm having a hard time drawing conclusive insights from these results. LLM+Caption wins in two columns, LLM+Caption+VQA wins in two columns, while LLM+Caption+LLMclue wins in 4 columns. None of the settings is consistently stronger. Nor do the results suggest a consistent way to choose settings based on the problem category.

Author response:
1. **The rationale for branching based on the problem category and the confidence.** This is an important question. In the decision of choosing the branch for solving the question, we consider two aspects: performance and efficiency, evaluating by the number of LLM calls.

    (a) From Table 1, we can observe that when the LLM is confident about its initial reasoning, its performance is superior or nearly the best for both VCU and VCI problems. In addition, [LLM w/ Caption] is the most compute-efficient. Therefore, using [LLM w/ Caption] is a **cost-effective** option. 

    (b) When the LLM is not confident about its initial reasoning on VCI problems, [LLM w/ Caption+LLM clue] significantly outperforms other decision paradigms. On VCU problems, we can observe that the performance of BLIP2 is similar to [LLM w/ Caption+LLM clue]. However, the [LLM w/ Caption+LLM clue] requires five LLM calls, which is three LLM calls more than using BLIP2. Therefore, using BLIP2-ITA is a **cost-effective** option in this case.

    We briefly mentioned the efficiency aspect in the original version. We’ve updated the explanation for the branching rationale in Section 6.1.

Reviewer comment (Experiments):
This paper only produced preliminary results on small validation sets and with a single set of decoding configurations. So, the results might lack generality and comprehensiveness. More comprehensive experiments across larger datasets and multiple seeds/decoding configurations would significantly strengthen the arguments the authors have sought to put forth.

Author response:
2. **Results on more examples and decoding configurations of VCR dataset.** Thank you for the suggestion. We agree running on more examples and more decoding configurations could better help validate our findings. Therefore, we sample **3000 examples** from the VCR dataset. The results are shown below: 
| Model           | VCU conf | VCU !conf | VCI conf | VCI !conf |
|-----------------|----------|-----------|----------|-----------|
| BLIP2-Pretrain  | 70.0     | 56.3      | 59.2     | 47.4      |
| BLIP2-Pretrain-clue | 70.6 | 56.7      | 63.3     | 49.2      |
| Caption             | 75.3     | 46.6          | 65.3     | 41.9      |
| Caption + VQA Clue | 75.9  | 51.9      | 65.3     | 47.3      |
| Caption + LLM Clue | 72.9  | 58.1      | 57.1     | 52.9      |

    As shown above, the conclusions drawn from Table 1, and the branching strategy explained above (and in the revision of the paper) remain unchanged compared with the results on 500 examples.

    We also run the **[Caption + LLM Clue]** version in Table 1 with extra temperatures and with different in-context examples in the prompt for thinking visual observations to support each choice (Fig.3 Right).

    | Caption + LLM Clue  | VCU conf | VCU !conf | VCI conf | VCI !conf |
    |-----------------|----------|-----------|----------|-----------|
    | Original        | 72.9     | 58.1      | 57.1     | 52.9      |
    | temp 0.1        | 72.4     | 58.8      | 57.1     | 53.9      |
    | Temp 0.2        | 72.4     | 58.5      | 61.2     | 52.2      |
    | ICL example     | 74.7     | 56.7      | 59.2     | 54.2      |

   We can also observe that, using different LLM temperature and in-context examples only have a small influence on the performance, and does not affect the conclusions. We’ve updated the results in the revision.

Reviewer comment (Novelty):
Existing methods, such as BLIP2, instructBLIP, and mini-GPT4, align the visual context with the LLM inputs embedding instead of input words and achieve better performance. It is not clear why the proposed method uses words (caption or VQA result) to transfer information from the VLM to LLM.

Author response:
1. **Why use words to transfer information between LLMs and VLMs.**
This is an important question. We acknowledge that relying solely on text for communication between LLMs and VLMs is a limitation of our study, a point we also mentioned in the limitations section. However, we believe our submission is a valuable addition to the literature due to following reasons:

    **(a)** Currently, connecting top-performing closed-source proprietary LLMs directly with perception (e.g. image, video, point cloud, etc.) modules through embeddings is challenging. As a result, the use of captions remains the only method for establishing this connection. Hence, most current research in multi-modal applications of LLMs primarily relies on caption-based approaches.

    **(b)** Utilizing text for communication between LLMs and VLMs provides a clear advantage in terms of interpretability. This interpretability is crucial for comprehending the behavior of our method. While embeddings could serve as an alternative medium, they tend to act as a "black-box", offering limited interpretability. This lack of clarity can lead to challenges in debugging and elongate the development process.

    **(c)** In our initial investigations, we observed a comparable effect when linking LLMs and VLMs through embeddings. Specifically, these models can neglect important visual cues in scenes characterized by clutter and occlusions because they do not actively search for visual cues based on the context of the scene and the question. One possible extension of our approach could involve implementing an attention mechanism over visual embeddings, guided by a high-level comprehension of the scene and contextual information, much like the methodology we have employed.

Reviewer comment (Reproducibility):
The paper lacks implementation details of the proposed model and compared methods. BLIP2 has a different setting to obtain the answer to the original one.

Author response:
2. **Implementation details of experiments and BLIP2 with LLM.** Thank you for the suggestion. We included the implementation details of the compared methods and our method in Sec. 5.2 and Sec. 5.3. We also add the results of the implementation of BLIP2 with LLM below and in the paper. 
| Model        | AOKVQA | VCR  |
|--------------|--------|------|
| BLIP2-Pretrain | 65.6   | 51.2 |
| BLIP2-T5xl    | 71.6   | 53.6 |
| ViCor (ours)       | **75.6**   | **59.8** | 

    As shown above, the BLIP2 with LLM version performs better than the version without LLM, which is mainly because the LLM of BLIP2 improves the commonsense reasoning capability based on the perceived information. Equipping BLIP2 with larger LLMs could potentially improve its performance on visual commonsense reasoning. Currently, connecting top-performing proprietary LLMs like GPT-4 directly with BLIP2 through embeddings is not possible. As a result, the use of captions remains the only method for this connection, underlining the importance of our approach.

Reviewer comment (Writing):
What does the j in o_j represent? How is the range of j determined?

Author response:
3. **Explanation of $j$ in $o_j$** As in Eq. 6, $o_j$ corresponds to the $j^{th}$ visual factor. The range of $j$, which is the number of visual factors, is determined by the visual factor reasoning performed by the LLM as in Fig. 2 (e.g. there could be one or multiple visual factors required to answer the question). The prompt for visual factor reasoning is in Fig. 3 middle.

Reviewer comment (Experiments):
How are the in-context examples obtained? From training set or manually written?

Author response:
4. **Construction of the in-context examples.** The questions in the in-context examples in the prompt are selected from the training set.

Reviewer comment (Experiments):
The BLIP-2 can directly generate the answer for visual questions. How does the BLIP-2 perform in this setting?

Author response:
2. **Implementation details of experiments and BLIP2 with LLM.** Thank you for the suggestion. We included the implementation details of the compared methods and our method in Sec. 5.2 and Sec. 5.3. We also add the results of the implementation of BLIP2 with LLM below and in the paper. 
| Model        | AOKVQA | VCR  |
|--------------|--------|------|
| BLIP2-Pretrain | 65.6   | 51.2 |
| BLIP2-T5xl    | 71.6   | 53.6 |
| ViCor (ours)       | **75.6**   | **59.8** | 

    As shown above, the BLIP2 with LLM version performs better than the version without LLM, which is mainly because the LLM of BLIP2 improves the commonsense reasoning capability based on the perceived information. Equipping BLIP2 with larger LLMs could potentially improve its performance on visual commonsense reasoning. Currently, connecting top-performing proprietary LLMs like GPT-4 directly with BLIP2 through embeddings is not possible. As a result, the use of captions remains the only method for this connection, underlining the importance of our approach.

Reviewer comment (Experiments):
The instructBLIP is not trained on the VCR dataset. What is its zero/few-shot performance on VCR?

Author response:
5. **Results of InstructBLIP on VCR.** Thank you for the suggestion. The structure of InstructBLIP doesn’t support few-shot **multi-modal** in-context learning. Therefore, we test the zero-shot result of InstructBLIP on VCR dataset, and the result is shown below. 
| Model             | VCR  |
|-------------------|------|
| InstructBLIP-Vicuna7b | 46.2 |
| InstructBLIP-T5xl    | 54.6 |
| ViCor (ours)            | **59.8** |

    From the result, we can see that instruction tuning on a large number of standard vision-and-language benchmarks improves the generalization abilities of VLMs to new tasks compared with BLIP2 using the same T5-XL LLM. However, the performance of InstructBLIP-Vicuna7b version is significantly worse, and our ViCor method still significantly performs better.

Reviewer comment (Experiments):
What is the LLM size of the compared baseline? Do they have a similar number of parameters to the proposed model?

Author response:
6. **LLM size of the compared baselines.** The number of parameters in GPT-3.5-turbo or GPT-4 is unknown. However, concurrent in-context-learning-based methods and chain-of-thought baselines listed in Table 2 and Table 3 use the same LLMs as ours. BLIP2 and InstructBLIP compared above are using T5XL with 3B parameters, and Vicuna with 7B parameters.

Reviewer comment (Experiments):
Experiments could be more exhaustive -- for instance, why not expand the experiments into more VCR datasets such as OKVQA (Marino et al.), VisualComet (Park et al.), V2C (Fang et al)?

Author response:
1. **The design motivation of our method and experiments on more datasets.** Thanks to the reviewer for mentioning these VQA datasets. We agree that a system that can improve all kinds of visual question answering would be the best. However, the motivation of the design of our framework is twofold: firstly, to leverage complementary strengths of VLMs and LLMs under VCU and VCI problems; and secondly, to enable let LLMs direct VLMs in gathering relevant visual elements that aid in making commonsense inferences. Therefore, our study and framework does not aim to improve the **grounding and spatial perception capabilities**, which general visual question answering benchmarks like VQA and GQA emphasize.
    
    As suggested, we apply our framework to another visual **commonsense reasoning** dataset, specifically OKVQA. It's important to highlight that OKVQA represents an **open-ended** question-answering (QA) benchmark, distinguishing it from the **multiple-choice** QA benchmarks presented in our original submission. We randomly sample 1000 examples from OKVQA dataset and test our framework on them, the results are shown below:
    | Model     | OKVQA |
    |-----------|-------|
    | LLM+Caption | 34.3  |
    | BLIP2-T5xl | 36.0  |
    | ViCor (ours)    | **38.4**  |

    Since OKVQA is an **open-ended** QA dataset, we use the Caption+VQA clue version of our method in Table 1 to tackle unconfident VCI problems. We use GPT-3.5 as the LLM modules in the experiments. As shown above, our framework can still leverage the complementary capabilities of both VLMs and LLMs to achieve better results owing to problem classification and active visual information acquisition.

Reviewer comment (Experiments):
The pipeline doesn't seem to be specific to VCR and could be used for any VQA dataset (eg. VQAv2, GQA, CLEVR, etc.) -- it's not clear whether the proposed method also improve performance on these datasets. In practice, questions to a real-time system could be of any type (those about commonsense or those about simple perception) -- so it would be important to improve performance on both.

Author response:
1. **The design motivation of our method and experiments on more datasets.** Thanks to the reviewer for mentioning these VQA datasets. We agree that a system that can improve all kinds of visual question answering would be the best. However, the motivation of the design of our framework is twofold: firstly, to leverage complementary strengths of VLMs and LLMs under VCU and VCI problems; and secondly, to enable let LLMs direct VLMs in gathering relevant visual elements that aid in making commonsense inferences. Therefore, our study and framework does not aim to improve the **grounding and spatial perception capabilities**, which general visual question answering benchmarks like VQA and GQA emphasize.
    
    As suggested, we apply our framework to another visual **commonsense reasoning** dataset, specifically OKVQA. It's important to highlight that OKVQA represents an **open-ended** question-answering (QA) benchmark, distinguishing it from the **multiple-choice** QA benchmarks presented in our original submission. We randomly sample 1000 examples from OKVQA dataset and test our framework on them, the results are shown below:
    | Model     | OKVQA |
    |-----------|-------|
    | LLM+Caption | 34.3  |
    | BLIP2-T5xl | 36.0  |
    | ViCor (ours)    | **38.4**  |

    Since OKVQA is an **open-ended** QA dataset, we use the Caption+VQA clue version of our method in Table 1 to tackle unconfident VCI problems. We use GPT-3.5 as the LLM modules in the experiments. As shown above, our framework can still leverage the complementary capabilities of both VLMs and LLMs to achieve better results owing to problem classification and active visual information acquisition.

Reviewer comment (Experiments):
In Table 1, it is unclear how each dataset is divided into two parts VCU and VCI for evaluation.

Author response:
2. **How to divide VCU and VCI problems in the experiments.** This is a good question. In the ablation study presented in Table 1, the division between VCU and VCI is based on the output from GPT-3.5-turbo. The prompt we used is shown in Figure 3. As shown in our response to Reviewer eVt8 (Evaluation of the reliability of the problem classification by LLMs), the problem category classification by LLMs aligns well with human categorization, ensuring reliability in identifying problem types. It is important to note that in real usages (results of Tables 2 and 3), LLMs will attempt to classify the question into either VCU or VCI problem **only if** it is not confident in the initial reasoning, as shown in Fig. 2.

Reviewer comment (Writing):
Please define "Sup" and "ICL" for readers -- the acronyms may be common in the active LLM research community, but you cannot assume that readers are part of this active community, especially since in-context learning is only a couple of years old.

Author response:
3. **Clearer definitions.** Thanks to the reviewer for the warm suggestion. We’ve defined these terms in our revision.