Paper: ExoViP: Step-by-step Verification and Exploration with Exoskeleton Modules for Compositional Visual Reasoning (ICLR-2024)

Reviewer comment (Reproducibility):
Detailed information about different prompts is necessary for a comprehensive comparison

Author response:
In our submission, we have demonstrated the exact prompt for self-correctness for all five tasks, Please refer to **Appendix G** for details. 
Specifically, for example, for compositional question answering, the prompt goes like:
`You are a ranker for a planner who use the candidate modules to answer a question: QUESTION, select the best solutions for answering the question \n candidate modules include: LOC: detection, VQA: visual question answering, EVAL: use logic operation, RESULT: wrap up the final result, CROP/CROP_LEFTOF/CROP_RIGHTOF/CROP_FRONTOF/CROP_INFRONT/CROP_INFRONTOF/CROP_BEHIND/CROP_AHEAD/CROP_BELOW/CROP_ABOVE: crop the image \n Current solutions: \n 0 PLAN \n 1 PLAN \n If the modules in the solutions have better cause-and-effect relations, and more likely to answer the question, please rank it first. If you are unsure, please keep the original rank. Return sequence number of currently best solution, for example 0,1,2,3, DO NOT RETURN ANYTHING ELSE EXCEPT FOR NUMBERS SPLIT by ,`

We are willing to append more details if there is any further missed information.

Reviewer comment (Evaluation):
The improvements on the GQA dataset appear to be minor when compared to those achieved by ViperGPT

Author response:
Thank you for bringing this to our attention. We would like to highlight 4 aspects: 

- Our test set differs from that of ViperGPT. We used the same sampling strategy as in VisProg. As is claimed in **Section 5.1**, `we randomly sample 5 samples from the balanced val set and 20 samples from the testdev set of each question type. e.g. “weatherVerify” for judging the weather, “twoCmomon” for judging common attributions of two objects. In summary, there are 102 question types and 2327 questions in our test set.`
- Regarding the different test sets, we re-executed ViperGPT's experimental results on our test set using their official repository (https://github.com/cvlab-columbia/viper) without any modifications. The results indicate that VisProg+ExoViP (61.49%) improves the performance of ViperGPT (46.47%) by 5.02%.
- Our insights differ from ViperGPT. While ViperGPT has transitioned to Python code for planning, thereby inherently addressing numerous planning errors, our module is designed to enhance existing compositional methods by tackling both planning and module errors.
- As shown in Table 7, ExoViP can be generalized to improve ViperGPT by 1.37%.

Reviewer comment (Experiments):
The verifier module is not carefully developed, CLIP for image-text matching is particularly not good at give correct similarity score, especially when the task are for compositional visual reasoning in this paper. In addition BLIP version1 (as cited in Section 4.1) for image caption/VQA is also not SOTA, and no ablation study is explored here to choose the best module for verifiers.

Author response:
**A:** Sorry for any confusion. We have carefully considered the selection of models for each verifier module. Specifically,
1. **Fair module comparisons.** We intentionally selected modules from VisProg's original modules. As noted in the implementation detail section of **Appendix E**, "` In order to validate the effectiveness of our method and eliminate the benefits of external knowledge such as more advanced vision-language models which are trained on larger datasets. Both operation modules and verification modules are selected from the same candidate neural module sets.`". 
2. **Trade-off between latency and performance.** As claimed in **Section 5.2.2**, "`while Qwen-VL is built on a LLM with 7 billion parameters, which is trained on trillions of tokens from the corpus, our method assembles a team of experts whose collective parameters total less than 1 billion`". Our goal is not only to achieve better performance with heavier and more models, but to deliver a carefully designed system that is both lightweight and easy to plug-and-play, offering superior performance with minimal additional computational expense. Specifically, when we substitute the QA model (blip-vqa-capfilt-large) with BLIPv2 (blip2-flan-t5-xl), we see a performance increase from 67.96 to 68.70 on NLVR. However, this comes with a *tenfold increase in parameters*.

Reviewer comment (Evaluation):
The introduced verifiers can cost the whole system more computation time, however no efficiency info is added here, it should be more fair to consider this. Besides, the sub-verifiers for calibration introduce propagation error which is not handled, and it's not clear when the verification score is more reliable than the confidence score, while in the method, the verification score directly replace it.

Author response:
**A:** Thank you for pointing it out. We present the mean inference time on the GQA dataset. Generally, the temporal expenditure of our tree-based search method significantly surpasses that of VisProg. However, the majority of the time is consumed by the call of the OPENAI API, an issue we posit is intrinsic to analogous works [1,2,3]. When compared to Depth First Search/ Breadth First Search [1] or Monte Carlo Tree Search [2,3], we assert that our beam search-based method can achieve an optimal equilibrium between efficiency and effectiveness. We have added this part to **Appendix D** in our revised paper.

| model   | Total Inference Time (s) |  Planning Time (s)   | Module Inference Time (s) |
| ------- | -------------------- | --- | -------------------------- |
| VisProg | 1.59                 |  1.10   | 0.49                       |
| ExoViP  | 4.32                 |   3.64  | 0.68                       |

Reviewer comment (Experiments):
Very concerned with the compared baselines. e.g., many recent MLLMs achieve better results on GQA than the baselines used in Table 1, e.g., LLaVA get 63 on it. In Table 3, why is the reported referring expression comprehension results only on RefCOCO and RefCOCO+, why not including RefCOCOg, which is quite a standard comparison for this task. And why only a subset is validated, 2 samples per type from the test set is actually not enough. And the compared model and proposed model are only 30 IoU, which is far below the decent results in recent MLLM, which all receive 80+ (e.g., Shikra, Kosmos2, etc)

Author response:
**A:** We would like to clarify the selection of both a subset of datasets and baseline models. 
- **Dataset:** As we mentioned in **Section 5.1**, "`becasue VISPROG doesn’t release their sampled evaluation subset, we do sampling following the VISPROG paper and evaluate all the methods on our sampled evaluation set.`" For a subset of the datasets, we follow the same rationale as VISPROG. "`In order to limit the money spent on generating programs with GPT-3, we create a subset of GQA
for evaluation.`" We also align with VISPROG in terms of the number of datasets we use. Specifically, for the Factual Knowledge Object Tagging task, they have "`100 tagging instructions`", and for the Image Editing task, they have "`107 instructions`".
- **Baselines:** Our insights are different with MLLMs. While most of them are designed to incorporate multimodal instructions into LLMs to achieve SOTA on QA tasks in general domain (e.g. [MME](https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation), [MMBench](https://opencompass.org.cn/leaderboard-multimodal), [SEED-Bench](https://huggingface.co/spaces/AILab-CVC/SEED-Bench_Leaderboard)),  our method is specifically designed to address the issues of existing visual compositional methods and to better integrate MLLMs into our framework. To clarify, in our experimental setting, we have not been able to find a single MLLM that is suitable for all five tasks. Consequently, we have to select parts of the SOTA models for reference. As we mentioned in **Section 1**, all the experiments are designed to "`show consistent improvements with the two models on the five tasks`". It's important to note that we have not claimed to achieve a state-of-the-art result in this work. In response to your request, we will endeavor to incorporate additional baseline results in the subsequent discussion.

Reviewer comment (Experiments):
In conclusion, the main concern is the baselines compared in each Table, very confusing, a lot SOTA models are removed out e.g., OFA-large is selected for NLVR, but not included for RefCOCO (which is actually reported in the origin paper). And the proposed method is not carefully developed or verified, as not sufficient ablation for each module is conducted, especially the choice for each verifier.

Author response:
**A:** As previously stated, our objective in this work is not to achieve the SOTA performance on all tasks. We have selectively referenced parts of the SOTA models. Since Qwen-vl-7b significantly outperforms OFA-Large on the referring task, we did not report its results.

Reviewer comment (Experiments):
For the three proposed verifiers, the CLIP is used for image-text matching, and BLIP is used for image captioning and VQA, how these models are selected, is there any ablation experiments here? E.g., C_img is generated by BLIP, why not using SOTA caption model, at least BLIPv2? Is this a severe bottleneck for the verifier or even bring confusion when the verifier model make errors.

Author response:
**A:** Sorry for any confusion. We have carefully considered the selection of models for each verifier module. Specifically,
1. **Fair module comparisons.** We intentionally selected modules from VisProg's original modules. As noted in the implementation detail section of **Appendix E**, "` In order to validate the effectiveness of our method and eliminate the benefits of external knowledge such as more advanced vision-language models which are trained on larger datasets. Both operation modules and verification modules are selected from the same candidate neural module sets.`". 
2. **Trade-off between latency and performance.** As claimed in **Section 5.2.2**, "`while Qwen-VL is built on a LLM with 7 billion parameters, which is trained on trillions of tokens from the corpus, our method assembles a team of experts whose collective parameters total less than 1 billion`". Our goal is not only to achieve better performance with heavier and more models, but to deliver a carefully designed system that is both lightweight and easy to plug-and-play, offering superior performance with minimal additional computational expense. Specifically, when we substitute the QA model (blip-vqa-capfilt-large) with BLIPv2 (blip2-flan-t5-xl), we see a performance increase from 67.96 to 68.70 on NLVR. However, this comes with a *tenfold increase in parameters*.

Reviewer comment (Theory):
It happens when the confidence score is more accurate than the verification score. However, seems in candidate prediction in Equation 6, is only using the averaged verification score to calibrate, while the confidence score is ignored.

Author response:
**A:** Sorry about the confusion. There might be severe misunderstandings regarding our method. we never used the verification score to replace the confidence score. In Equation 6, $p$ represents the confidence score, while $w$ denotes the normalized verification score, also known as the weight. Intuitively, the verification score, or weight, is used to scale (calibrate) the confidence score. We have not ignored the confidence score at any point.

Reviewer comment (Theory):
How is the propagation error being controlled? The verifier is proposed to mitigate the planning error and module error. How is the newly. introduced verifier module error being handled?

Author response:
**A:** Apologies for any confusion. There are two key components in our method specifically designed to mitigate the problem.

- **Negative Sampling Strategy**: We calculate the verification score as the difference between candidates and their negative samples. This score serves as a verification score and is used to calibrate the confidence score. As outlined in **Section 4.1**, we discovered that without this strategy,` directly applying this verification score does not work well.` Therefore, `Motivated by recent works in truthfulness [4], commonsense [5], and bias [6], ` We proposed the negative sampling strategy. Our method is also partially inspired by recent developments in multimodal counterfactuality [7, 8]. The experimental results shown in Table 1 further demonstrate the effectiveness of this negative sampling strategy. In other words, we adopt negative sampling strategy to prevent propagation errors introduced by verifiers.
- **Combination of Verification Score and Self-correctness of LLMs**: To improve planning results as well as the robustness of our method, we incorporate the verification score with self-correctness of LLMs in the tree-based search algorithm. As claimed in **Section 4.2**, `because the verification scores can be very close for the selected K traces, we further use the self-correctness ability of LLMs to reorder the K traces and select the top P from them (P < K).` The experimental results shown in Table 1 further demonstrate the effectiveness of the combination. In other words, we leverage the self-correctness of LLMs to prevent propagation errors introduced by verifiers.

Reviewer comment (Evaluation):
The initial analysis finds the percentage of errors caused by the two types of problems (module error and planning error). The developed method's intuition is to solve these two errors with the two mentioned strategies. However, the final analysis does not include the kind of errors this method actually ends up fixing and in what ratio. It would be beneficial to show the statistics that demonstrate what proportion of errors of each type were fixed by the proposed methods compared to VISPROG [1].

Author response:
**A:** Thank you for the valuable suggestion. In our original submission, we have manually analyzed 100 randomly sampled failure cases on VisProg. We find that there are three typical reasons for the failures: a) vision module prediction error; b) LLM planning error; c) others. We demonstrate the statistics of the failure cases in **Appendix.C**. We provide illustrations of two representative failure cases resulting from our methodology in **Appendix I**.

In our revised paper, following the application of our proposed framework, we re-assessed the same cases. As shown in **Appendix C**, "`we were pleased to discover a reduction in module errors by 28.87%, and a decrease in planning errors by 42.35%. Nevertheless, our current strategy was unable to rectify 69.8% of the errors. When juxtaposed with the data from Table 1, our method has enhanced VISPROG by 7.11%, which is lower than the improvement of the failure cases. This outcome suggests that our approach may give rise to novel challenges. We further demonstrate common errors of our method in Figure 10 and Figure 11, As highlighted in the study of failure cases, the majority of these instances originate from the VQA module.`"

Reviewer comment (Experiments):
The method is not evaluated with any open-source Large Language Models (LLaMA, Vicuna, etc.) Restricting evaluation only to GPT-3.5-turbo reduces the generalizability of the approach.

Author response:
**A:** Thank you for the helpful suggestion. In fact, both the official implementations of [ViperGPT](https://github.com/cvlab-columbia/viper) and [VISPROG](https://github.com/allenai/visprog) are supported by the GPT-3.5-turbo. In accordance with this line of research, all the experimental results reported herein are also built on the GPT-3.5-turbo model. In addition, we replaced the GPT-3.5-turbo with LLama2-chat-13b and obtained the following results. We are thrilled to discover that our method can yield significant improvements in open LLM. We have added these results to **Section 5.3** in our revised paper.


| model | accuracy on GQA | 
| -------- | -------- |
| VISPROG (GPT-3.5-turbo)     | 57.41     |
| ExoViP (GPT-3.5-turbo)     | 61.49     |
| VISPROG (Llama-2-13b-chat)    | 46.41     |
| ExoViP (Llama-2-13b-chat)     | 54.45     |

Reviewer comment (Evaluation):
The performance increment across tasks over VISPROG [1] is still incremental, considering the time cost of using beam search and multiple verification modules.

Author response:
**A:** Thank you for the constructive advice. We present the mean inference time on the GQA dataset. Generally, the temporal expenditure of our tree-based search method significantly surpasses that of VisProg. However, the majority of the time is consumed by the call of the  OPENAI API, an issue we posit is intrinsic to analogous works [3,4,5]. When compared to Depth First Search/ Breadth First Search [3] or Monte Carlo Tree Search [4,5], we assert that our beam search-based method can achieve an optimal equilibrium between efficiency and effectiveness. We have added this part to **Appendix D** in our revised paper.

| model   | Total Inference Time (s) |  Planning Time (s)   | Module Inference Time (s) |
| ------- | -------------------- | --- | -------------------------- |
| VisProg | 1.59                 |  1.10   | 0.49    |
| ExoViP  | 4.32                 |   3.64  | 0.68         |

Reviewer comment (Evaluation):
In section 4.2 - You sort the candidate neural modules according to verification scores. From what your paper says verification scores are obtained for different candidates generated by a single neural module. Are these verification scores for the neural modules themselves different or are they the verification score of the top candidate of each neural module?

Author response:
**A:** Sorry about the confusion. In the tree-based search algorithm, the verification score for a single node is calculated as the average score of the top candidate. The accumulated score of the candidate path is then used for sorting purposes. We have clarified this part in **Section 4.2** of our revised paper.

Reviewer comment (Evaluation):
What is the inference latency for your approach compared to using VISPROG [1]? The use of beam search and multiple verification modules indicates a significant added overhead.

Author response:
**A:** Thank you for the constructive advice. We present the mean inference time on the GQA dataset. Generally, the temporal expenditure of our tree-based search method significantly surpasses that of VisProg. However, the majority of the time is consumed by the call of the  OPENAI API, an issue we posit is intrinsic to analogous works [3,4,5]. When compared to Depth First Search/ Breadth First Search [3] or Monte Carlo Tree Search [4,5], we assert that our beam search-based method can achieve an optimal equilibrium between efficiency and effectiveness. We have added this part to **Appendix D** in our revised paper.

| model   | Total Inference Time (s) |  Planning Time (s)   | Module Inference Time (s) |
| ------- | -------------------- | --- | -------------------------- |
| VisProg | 1.59                 |  1.10   | 0.49    |
| ExoViP  | 4.32                 |   3.64  | 0.68         |

Reviewer comment (Experiments):
Both ViperGPT [2] and VISPROG [1] implementations used GPT-3 as the model of choice. Will the advantages of your method transfer to GPT-3 and other Open LLMs as well? Or is GPT-3.5-turbo necessary in order to accomplish Self-Correction and Trace Reasoning?

Author response:
**A:** Thank you for the helpful suggestion. In fact, both the official implementations of [ViperGPT](https://github.com/cvlab-columbia/viper) and [VISPROG](https://github.com/allenai/visprog) are supported by the GPT-3.5-turbo. In accordance with this line of research, all the experimental results reported herein are also built on the GPT-3.5-turbo model. In addition, we replaced the GPT-3.5-turbo with LLama2-chat-13b and obtained the following results. We are thrilled to discover that our method can yield significant improvements in open LLM. We have added these results to **Section 5.3** in our revised paper.


| model | accuracy on GQA | 
| -------- | -------- |
| VISPROG (GPT-3.5-turbo)     | 57.41     |
| ExoViP (GPT-3.5-turbo)     | 61.49     |
| VISPROG (Llama-2-13b-chat)    | 46.41     |
| ExoViP (Llama-2-13b-chat)     | 54.45     |

Reviewer comment (Evaluation):
Could you report an analysis about what percentage of module errors and planning errors were fixed by your method compared to VISPROG?

Author response:
**A:** Thank you for the valuable suggestion. In our original submission, we have manually analyzed 100 randomly sampled failure cases on VisProg. We find that there are three typical reasons for the failures: a) vision module prediction error; b) LLM planning error; c) others. We demonstrate the statistics of the failure cases in **Appendix.C**. We provide illustrations of two representative failure cases resulting from our methodology in **Appendix I**.

In our revised paper, following the application of our proposed framework, we re-assessed the same cases. As shown in **Appendix C**, "`we were pleased to discover a reduction in module errors by 28.87%, and a decrease in planning errors by 42.35%. Nevertheless, our current strategy was unable to rectify 69.8% of the errors. When juxtaposed with the data from Table 1, our method has enhanced VISPROG by 7.11%, which is lower than the improvement of the failure cases. This outcome suggests that our approach may give rise to novel challenges. We further demonstrate common errors of our method in Figure 10 and Figure 11, As highlighted in the study of failure cases, the majority of these instances originate from the VQA module.`"