Paper: RelationVLM: Making Large Vision-Language Models Understand Visual Relations (ICLR-2024)

Reviewer comment (Reproducibility):
The authors introduced how to construct data for RelationVLM but did not explain the overall training. Figure 2 shows the overall training pipeline, but there are no sections or sentences that refer to Figure 2.

Author response:
Thanks for this suggestion. We add overall training pipeline to Sec. 3.3 that refers to Figure 2 in our revision. Besides, the detailed descriptions for each training stage are also presented in this section.

Reviewer comment (Evaluation):
Relation Score is a new evaluation metric based on the assessment from an LLM. However, the evaluation cannot rely on an LLM as RelationVLM was trained based on a dataset constructed based on an LLM-based decoder.

Author response:
In fact, evaluating the performance of large-scale models remains a highly challenging issue currently. We employ a LLM-based assessor for the model evaluation, following the current common practices [1][2][3] in this field. We will explore this aspect in our future study.

Reviewer comment (Writing):
Section 3.1: What is $N$ in $\mathcal{D} = \{(x_i , y_i )\}_{i=0}^N$ ?

Author response:
Sorry for this confusion. $N$ is the number of training samples. We add this description in the revision.

Reviewer comment (Writing):
Page 5: $\mathcal{R}_{n1}(\cdot)$ any typo?

Author response:
Sorry for this confusion. But we confirm that there is no typo in $R_{n1}(\cdot)$.

Reviewer comment (Experiments):
The paper could enhance its validation of RelationVLM by extending the range of benchmarks and comparisons with existing models, particularly those that are considered state-of-the-art in the field of vision-language models. This would provide a more solid foundation for assessing the model's performance and capabilities.

Author response:
**Q1: Extending the range of benchmarks and comparisons with existing models.**  
**A1:** Thanks for this suggestion. We have performed a broader comparison over extended benchmarks with more models. The corresponding results are **in Table 5 of our revision**.

Reviewer comment (Experiments):
The process of data construction is central to the training of RelationVLM, yet the paper does not delve into potential biases that might be introduced during this phase. A thorough analysis of data diversity and strategies to mitigate bias would contribute to the robustness and reliability of the model.

Author response:
**Q2: A thorough analysis of data diversity and strategies to mitigate bias would contribute to the robustness and reliability of the model.**  
**A2:** Thank you for this valuable suggestion. Currently, the diversity and bias of our data depend to some extent on GPT-4, which is recognized as the most advanced large-scale model. In fact, evaluating the performance of large-scale models, including the diversity and bias of their data, remains a highly challenging issue. We have not yet found a scientific evaluation method from existing publications, and will explore this issue in our future research.

Reviewer comment (Reproducibility):
The complexity of the model architecture and training scheme necessitates a discussion on the computational resources required, as well as the scalability and efficiency of the model across different settings and applications.

Author response:
**Q3: The complexity of the model architecture and training scheme necessitates a discussion on the computational resources required, as well as the scalability and efficiency of the model across different settings and applications.**  
**A3:** We detail the complexity analysis as follows. The vision backbone has 1B parameters, and the vicuna model we used has 7B parameters, totally number of trainable parameters is 0.5B.  In the first stage, we conduct training over 100,000 steps using 4$\times$A100 GPUs, with a global batch size of 96 and a base learning rate of 2e-3. This stage is completed in approximately 30 hours. In the second stage, the model is trained for 20,000 steps on 4$\times$A100 GPUs, maintaining a global batch size of 96 and a base learning rate of 2e-4, taking around 6 hours. Finally, the model undergoes an additional 5,000 training steps on 4$\times$A100 GPUs, with a global batch size of 64, completed in about 1.5 hour. The maximum learning rate remains constant at 2e-5 during this final stage. We have added this analysis in our revised supplementary materials.

Reviewer comment (Writing):
The paper aims to enhance the model's comprehension of visual relations, but the definitions and explanations of these relations are somewhat concise. Expanding on the types of visual relations, along with providing more examples, would offer clearer insights into the model's understanding and categorization of these relations.

Author response:
**Q4: The paper aims to enhance the model's comprehension of visual relations, but the definitions and explanations of these relations are somewhat concise. Expanding on the types of visual relations, along with providing more examples, would offer clearer insights into the model's understanding and categorization of these relations.**  
**A4:** Thanks for this valuable suggestion as well. Our work is the first one of its kind to explore how to enable the capabilities of understanding visual relations for LVLMs. As an initial study, we start with the three most basic and widely applied high-level categories of visual relations. Here, we further clarify their definitions. For the first class, semantic relation, there are the similarity and the contrast. Among them, the similarity refers to the degree for a given object of being similar to the other one in another image in terms of their category. If they are of the same category, the model should ground them spatially. The contrast is defined as the difference between two instances in terms of their category as well as a series of semantic attributes. The temporal association and the geometric transform refer to the motions or transforms over time and in spatial, respectively. We will attempt to expand the types of visual relations in our future investigation.

Reviewer comment (Experiments):
The evaluations presented primarily focus on controlled settings. Incorporating assessments of the model's performance in real-world scenarios would demonstrate its applicability and effectiveness outside of experimental conditions.

Author response:
**Q5: The evaluations presented primarily focus on controlled settings. Incorporating assessments of the model's performance in real-world scenarios would demonstrate its applicability and effectiveness outside of experimental conditions.**  
**A5:** Our in-context learning evaluation on anomaly detection or medical diagnosis could be viewed as in-the-wild assessment in real-world scenarios, as their corresponding data and tasks are unseen for the model training. Besides, we will release a model demo for more assessments in the wild once completing our internal reviews.

Reviewer comment (Experiments):
Could you provide more information on the choice of benchmarks for evaluating RelationVLM? Including additional benchmarks, especially those involving state-of-the-art models, could strengthen the validation of RelationVLM's capabilities.

Author response:
**Q1: Extending the range of benchmarks and comparisons with existing models.**  
**A1:** Thanks for this suggestion. We have performed a broader comparison over extended benchmarks with more models. The corresponding results are **in Table 5 of our revision**.

Reviewer comment (Experiments):
How does the data construction process account for potential biases, and what steps were taken to ensure data diversity? A detailed explanation would enhance the robustness of the model and ensure the generalizability of the results.

Author response:
**Q2: A thorough analysis of data diversity and strategies to mitigate bias would contribute to the robustness and reliability of the model.**  
**A2:** Thank you for this valuable suggestion. Currently, the diversity and bias of our data depend to some extent on GPT-4, which is recognized as the most advanced large-scale model. In fact, evaluating the performance of large-scale models, including the diversity and bias of their data, remains a highly challenging issue. We have not yet found a scientific evaluation method from existing publications, and will explore this issue in our future research.

Reviewer comment (Reproducibility):
Can you elaborate on the computational resources required for RelationVLM, and discuss its scalability and efficiency across different settings?

Author response:
**Q3: The complexity of the model architecture and training scheme necessitates a discussion on the computational resources required, as well as the scalability and efficiency of the model across different settings and applications.**  
**A3:** We detail the complexity analysis as follows. The vision backbone has 1B parameters, and the vicuna model we used has 7B parameters, totally number of trainable parameters is 0.5B.  In the first stage, we conduct training over 100,000 steps using 4$\times$A100 GPUs, with a global batch size of 96 and a base learning rate of 2e-3. This stage is completed in approximately 30 hours. In the second stage, the model is trained for 20,000 steps on 4$\times$A100 GPUs, maintaining a global batch size of 96 and a base learning rate of 2e-4, taking around 6 hours. Finally, the model undergoes an additional 5,000 training steps on 4$\times$A100 GPUs, with a global batch size of 64, completed in about 1.5 hour. The maximum learning rate remains constant at 2e-5 during this final stage. We have added this analysis in our revised supplementary materials.

Reviewer comment (Writing):
The paper briefly explains different types of visual relations. Could you provide a more detailed taxonomy and additional examples to offer clearer insights into how the model comprehends and categorizes these relations?

Author response:
**Q4: The paper aims to enhance the model's comprehension of visual relations, but the definitions and explanations of these relations are somewhat concise. Expanding on the types of visual relations, along with providing more examples, would offer clearer insights into the model's understanding and categorization of these relations.**  
**A4:** Thanks for this valuable suggestion as well. Our work is the first one of its kind to explore how to enable the capabilities of understanding visual relations for LVLMs. As an initial study, we start with the three most basic and widely applied high-level categories of visual relations. Here, we further clarify their definitions. For the first class, semantic relation, there are the similarity and the contrast. Among them, the similarity refers to the degree for a given object of being similar to the other one in another image in terms of their category. If they are of the same category, the model should ground them spatially. The contrast is defined as the difference between two instances in terms of their category as well as a series of semantic attributes. The temporal association and the geometric transform refer to the motions or transforms over time and in spatial, respectively. We will attempt to expand the types of visual relations in our future investigation.

Reviewer comment (Experiments):
Are there evaluations of RelationVLM in real-world scenarios or applications?

Author response:
**Q5: The evaluations presented primarily focus on controlled settings. Incorporating assessments of the model's performance in real-world scenarios would demonstrate its applicability and effectiveness outside of experimental conditions.**  
**A5:** Our in-context learning evaluation on anomaly detection or medical diagnosis could be viewed as in-the-wild assessment in real-world scenarios, as their corresponding data and tasks are unseen for the model training. Besides, we will release a model demo for more assessments in the wild once completing our internal reviews.

Reviewer comment (Evaluation):
How does RelationVLM handle ambiguous or unclear visual relations in images or videos?

Author response:
**Q7: How does RelationVLM handle ambiguous or unclear visual relations in images or videos?**  
**A7:** As an initial exploration for teaching LVLMs to understand visual relations, we make our covered relation as basic and widely used as possible to ensure the generalization ability in different scenarios. For ambiguous or unclear visual relations that beyond our modelling range, frankly speaking, we are currently unable to make sure how the model performs. This may require case-by-case discussions, and is worthy of a further exploration in the future.

Reviewer comment (Experiments):
After works like "When and why vision-language models behave like bags-of-words, and what to do about it?", it is well-known that VLMs are weak in relation detection. Afterwards, there have been a few works in this domain. It is absolutely crucial to compare the proposed methods against (simple extension of) existing methods.

Author response:
**Q1: Comparison with CLIP-based VLMs like "When and why vision-language models behave like bags-of-words, and what to do about it?"**  
**A1:** Thank you for this suggestion. The work you list belongs to CLIP-based VLMs. We would like to point out that it's actually diffcult to compare ours with them on most evaluation tasks we used since they lack the capability to handle language input prompts to execute multiple different relation tasks, respond with language output to describe relations, or perform grounding to locate relations. Despite this, we still make a preliminary comparison with them in a comparable experimental setting. Specifically, we evaluate these CLIP-based VLMs in our contrast task using a method similar to the one we used to perform the comparisons with traditional reid models in our experiments. The comparison results are reported as follows:  
| Models   | CLIP[1] | NegCLIP[2] | VFC[3] | RelationVLM(Ours) |
|----------|:-------:|:----------:|:------:|:-----------------:|
| Acc (\%) | 60.5    | 65.4       | 66.3   | **83.2**          |

Reviewer comment (Presentation):
What is the baseline approach in Table 1? Is it Shikra or Kosmos-2?

Author response:
**Q3: What is the baseline approach in Table 1? Is it Shikra or Kosmos-2?**  
**A3:** The results in Table 1 correspond to comparisons at the model level, instead of an ablation study. Both Shika and Kosmos-2 should not be viewed as our baseline models in the ablation study.

Reviewer comment (Evaluation):
Why do you think anomaly detection is a good benchmark for showcasing the benefit from this approach? Please explain the link between anomaly detection and relation understanding.

Author response:
**Q4: Please explain the link between anomaly detection and relation understanding.**  
**A4:** We apply the task of anomaly detection for evaluating the capability of in-context learning with a few-shot evaluation setting. In this setting, as shown in Figure 5, normal and anomaly samples are given as references. When performing anomaly detection for a given sample, a visual comparison is necessitated between the test sample and the reference samples, in which visual relations are required to be captured for completing such comparison.

Reviewer comment (Experiments):
The baseline comparison is not enough. There are still many other strong LVLMs are not considered, such as LLaVA, MiniGPT-4, mPLUG-OWL, etc. Besides, why some tables have different comparisons? Some take KOSMOS as a baseline, others take Openflamingo as a baseline, which is quite confusing to me and is not a fair comparison.

Author response:
**A4:** In this work, we aim to enable the LVLMs with new capabilities in understanding visual relations that these model does not have. Thus, when we evaluate these target new capabilities, these LVLMs cannot deliver comparable results with ours for specific comparison. We add their corresponding evaluation results **in Table 5 of our revision**. These experimental results indicate that these models are very deficient in these capabilities. Besides these results, please also not that we have considered the comparisons with these models on the POPE benchmark un Table 8 of our supplementary since they are comparable on this benchmark.  

**A5:** We evaluate different capabilities by performing the corresponding comparisons in different tables. To ensure the fairness of the comparison, we only compare our model with the ones that use the same training data in these aspects.

Reviewer comment (Theory):
How to ensure the rationality of the geometric transformation?

Author response:
**Q3: How to ensure the rationality of the geometric transformation?**  
**A3:** As an initial exploration, we have not found off-the-shelf annotated datasets to use regarding geometric transform. Therefore, we segment natural images and apply random basic geometric transformations to the segmented objects for synthesizing the corresponding data. Although the dataset is synthesized, we strive to maintain a broad diversity of the synthesized data to enhance the generalization capability of the model as much as possible.

Reviewer comment (Presentation):
Better illustration is needed.

Author response:
**Q4: Better illustration.**  
**A4:** Thank you for your valuable suggestions. In the revision, we simplify the superscript and subscript in Eq.1 and add  Figure 3 for visualizing the data construction. Besides, we also highlight the best results in experimental results in bold as you suggest.

Reviewer comment (Writing):
In Eq. 1, the superscript and subscript are all too complex.

Author response:
**Q4: Better illustration.**  
**A4:** Thank you for your valuable suggestions. In the revision, we simplify the superscript and subscript in Eq.1 and add  Figure 3 for visualizing the data construction. Besides, we also highlight the best results in experimental results in bold as you suggest.

Reviewer comment (Presentation):
The data curation process needs a visualized process.

Author response:
**Q4: Better illustration.**  
**A4:** Thank you for your valuable suggestions. In the revision, we simplify the superscript and subscript in Eq.1 and add  Figure 3 for visualizing the data construction. Besides, we also highlight the best results in experimental results in bold as you suggest.

Reviewer comment (Presentation):
The best results in the tables can be bold.

Author response:
**Q4: Better illustration.**  
**A4:** Thank you for your valuable suggestions. In the revision, we simplify the superscript and subscript in Eq.1 and add  Figure 3 for visualizing the data construction. Besides, we also highlight the best results in experimental results in bold as you suggest.

Reviewer comment (Evaluation):
Though there were several tables of results, are their scale and generalization enough to support the claim?

Author response:
**Q5: On scaling and generalization performance of experimental results.**  
**A5:** Given that the evaluation of large-scale models is still a problem worth studying, here, we attempt to conduct some empirical experiments to further analyze the scaling and generalization performance of our experiment results. In terms of the scaling performance, we scale up the test set for evaluating the capability of grounding the objects of the same category to five times its original size. Its corresponding test results are reported in the table below. We can observe that our proposed RelationVLM consistently outperforms other LVLMs on this scaled test set. Regarding the generalization performance, we provide the few-shot results evaluated in the wild on real-world scenarios such as anomaly detection and medical diagnosis, in Figures 5, 6, and Table 3.  
| Method | BBox Acc (\%) | RS           |
|-----------------------------|:-------------:|:------------:|
| MiniGPT-4                   | -             | 1.3          |
| LLaVA                       | -             | 1.5          |
| MM-GPT                      | -             | 1.6          |
| mPLUG-Owl                   | -             | 1.7          |
| Shikra                      | 43.3          | 1.8          |
| Kosmos-2                    | 42.4          | 1.7          |
| RelationVLM(Ours)  | **49.6** | **2.6** |

Reviewer comment (Writing):
There are many controversies, but care is needed when using the word AGI, especially without discussion of the path to its precise description/definition and the relation between this work and AGI.

Author response:
**Q6: Be careful about using the word AGI.**  
**A6:** Thank you for the reminder. We have refined relavant statements by removing this word in the revision. We aim to narrow the gap between the capabilities of existing models and the anticipated AGI ambitions by compensating for the capabilities of existing LVLMs.

Reviewer comment (Presentation):
Though it is just a case, in Fig. 1, the shadow also differs.

Author response:
**Q7: In Fig. 1, the shadow also differs.**  
**A7:** Yes, this is a failure case of ours. As the first attempt to explore how to enhance LVLMs' understanding of visual relevance, our model has not yet reached the human level. And there are still some failures on challenging samples that have subtle differences.

Reviewer comment (Experiments):
Possible testing on visual relation understanding within one image/frame? Like two objects/persons in an image.

Author response:
**Q8: About the performance of visual relation understanding within one image/frame.**  
**A8:** Our proposed model can support the visual relation understanding within one image/frame. To demonstrate this, we conduct a validation experiment by combining two images of pedestrians into one and input it into the model for testing the accuracy of judging whether the two people on the left and right are the same person. The results are as follows:  
| Models   | MiniGPT-4 | LLaVA | RelationVLM(Ours) |
|----------|:---------:|:-----:|:-----------------:|
| Acc (\%) | 59.3      | 57.4  | **80.2** |

Reviewer comment (Evaluation):
In Tab 6, the Rec and Yes Ratio show disadvantages; any discussion?

Author response:
**Q9: In Tab 6, the Yes Ratio and Rec. show disadvantages, any discussion?**  
**A9:** It should be noted that the Yes Ratio is not necessarily better when it's higher or lower. Thus, the results do not indicate disadvantages. We follow [1] to measure the bias degree of our proposed model with this metric on its public benchmark in which the answers Yes and No account for half each. So a Yes Ratio near 50\% indicates a less biased model. Here, our RelationVLM outperforms other LVLMs in this regard. Besides, a lower Rec. (Recall) do not indicate the disadvantages as well. Balancing recall with precision is crucial to the effectiveness of any system. Overemphasizing one aspect can compromise the other. For instance, our model has a lower recall compared to LLaVA, which is a result of LLaVA's near 100\% Yes Ratio. This implies LLaVA's inclination to answer 'Yes' to all questions, which may inflate its recall, but doesn't necessarily indicate enhanced performance, considering its lower accuracy, precision, and F1 score. On the other hand, our model excels in both accuracy and F1 scores, demonstrating its overall superior performance. Therefore, despite the lower recall, our model's balanced performance metrics signify its effectiveness.  

We look forward to your feedback on our responses. Please feel free to let us know if you have any new questions or suggestions.

[1] Li, Yifan, et al. "Evaluating object hallucination in large vision-language models." In EMNLP, 2023.

Reviewer comment (Novelty):
The architecture of the proposed method, RelationVLM, is trivial. The core contribution of this paper may be a dataset contribution scheme. However, in the title and abstract, I cannot see a description of this core contribution.

Author response:
**Q1: Regarding highlighting the contributions on data construction.**  
**A1:** Thanks for your suggestion in this regard. We indeed compensate for the deficiency of current large vision-language models in understanding visual relations, based on commonly used model architectures. We have highlight this in the revision, marked in red.

Reviewer comment (Experiments):
The experiments may be not sufficient. If the visual relations comparison could benefit other visual tasks, e.g., image retrieval or fine-grain classification, such experiments should be conducted to further prove the meaning and value of the fine-grain relation or difference comparison of two images.

Author response:
**Q2: Could the visual relation comparison benefit other visual tasks, e.g., image retrieval or fine-grained classification?**  
**A2:** Thank you for this question. We conduct a fine-grained classification experiment on CUB200-2011 dataset, main results of which are presented in the following table. We add these results in Table 6 and illustrate a representative case study in Figure 8 of our revised version. These results demonstrate that our proposed RelationVLM outperforms SOTA LVLMs and some traditional task-specific models by a clear margin. 

| Models   | Part RCNN | PS-CNN | Bilinear-CNN | MiniGPT-4 | LLaVA | RelationVLM (Ours)          |
|----------|:---------:|:------:|:------------:|:---------:|:-----:|:--------:|
| Acc (\%) | 76.4      | 76.6   | 85.1         | 20.3      | 19.4  | **85.7** |