Paper: CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding (ICLR-2024)

Reviewer comment (Novelty):
**Fundamental Oversimplification of Compositionality:** The Compositional VLM framework, though innovative, may not truly capture the essence of disentangling objects and their relationships within images. Instead of delving deep into the inherent complexities of this challenge, the method leans towards gathering object-text-image paired datasets and reinforcing their connections. This approach, while seemingly effective, might only be a surface-level solution rather than addressing the root of the compositional problem.

Author response:
> **Q3. Fundamental oversimplification of compositionality.**

Bringing compositionality to VLM is a challenging problem, and we are the very first step to handle this challenge. The good results in representative compositional reasoning benchmarks show the effectiveness of our method in improving the compositional reasoning ability of VLMs, and we acknowledge that there are some limitations in our method such as not coping much with object-attribute compositionality and spatial event compositionality, which could be crucial future directions.

Reviewer comment (Reproducibility):
**Rigid Token Implementation**: The manual nature of token positioning (with <obj> around the object and <box> after them), suggests a lack of flexibility in the model. This rigidity could hamper the model's adaptability, especially when faced with varied or unforeseen testing scenarios. For example, if we have a "a man between two dog", how to deal with the <box> for multiple instance of dog.

Author response:
> **Q4. Rigid token implementation. If we have a "a man between two dog", how to deal with the <box> for multiple instances of dog?**

Our model can deal with the situation that there are multiple objects in an image, thanks to our detector (i.e., a YOLO-style model) can localize them simultaneously. 
    
Taking the text "*a man between two dogs*" as an example, a *<visual>* token is automatically generated after the word "dogs". Then, the object detector takes this token as input and outputs two boxes, each for one dog. All the features of predicted boxes will be sent back to LLMs, using one *<box>* for each detected box. Thus, the updated text would be "*a man between two \<obj\>dog\</obj\><visual><box><box>*". This gives our model the maximum flexibility to handle various scenarios.

Reviewer comment (Reproducibility):
**Operational Inefficiency**: The necessity for pre-parsing sentences and manually inserting tokens, even during testing, indicates potential operational bottlenecks. This could impede real-time applications and demands additional preprocessing steps, detracting from the model's overall efficiency.

Author response:
> **Q5. The necessity for pre-parsing sentences and manually inserting tokens, even during testing, indicates potential operational bottlenecks.**

Sorry for the confusion. Actually, all communication tokens are **automatically** generated by LLM. During the auto-regressive generation, users do not need to pre-parsing sentences nor manually insert communication tokens. We have provided more details about communication token generation in G1.

Reviewer comment (Evaluation):
When integrating the object detection model within the auto-regressive language model, what would be the time complexity for each individual inference? Would it be very slow?

Author response:
> **Q1. What would be the time complexity for each individual inference? Would it be very slow?**

In our implementation, we use the YOLOX detection head as our object detection model, which is very lightweight compared to the auto-regressive language model and CLIP vision encoder. The number of parameters for the object detection model is negligible compared to the whole model.
    
|  | #Params | Latency |
|:---:|:---:|:---:|
| Object detector | 3.4M | 2.2 ms |
| Vision encoder + LLM | 1,717.8M | 103.9 ms |
| Percentage | 0.2% | 2.1% |



    
We also measure the latency for the object detector and the vision encoder + LLM, separately. From the table above, the object detector takes only 2.1% of the total latency. Since the object detection model is not activated every time during training and inference, it will take less than 2.1\% time when we generate a sequence of tokens.

Reviewer comment (Experiments):
Given that the proposed Compositional VLM necessitates text parsing even during inference, how resilient is the model to inaccuracies or ambiguities in token placement? For instance, the phrase "a black dog" could be parsed as "a <obj>black dog<\obj>" or "a black <obj>dog<\obj>". How does the model handle such variations?

Author response:
> **Q2. How resilient the model is to inaccuracies or ambiguities in token placement.**

Our model is resilient to inaccuracies or ambiguities in token placement. We evaluate the token placement accuracy and our model's resilience to inaccurate token placement on RefCOCOg/RefCOCO+/RefCOCO datasets.
* Token placement accuracy measures how accurate *\<obj\>* and *\</obj\>* is placed compared to the ground truth. We handcraft several templates to contain the referring expression into a sentence. For example, one template is "I can see {expression} in the image". If both *\<obj\>* and *\</obj\>* are placed correctly, we count it as a true positive in overall accuracy. The placement accuracy for *\<obj\>* and *\</obj\>* are also separately evaluated.
* Localization accuracy measures the correctness of the predicted bounding box using either ground truth placement or predicted placement for *\<obj\>* and *\</obj\>*.

|  | RefCOCOg val | RefCOCO+ val | RefCOCO val |
|---|:---:|:---:|:---:|
| **Token Placement Accuracy** |  |  |  |
| Overall | 13.21 | 47.43 | 44.30 |
| <obj> | 99.84 | 99.84 | 99.85 |
| </obj> | 13.21 | 47.43 | 44.30 |
| **Localization Accuracy** |  |  |  |
| Predicted Placement | 58.74 | 45.77 | 45.71 |
| Ground Truth Placement | 60.87 (+2.13) | 47.62 (+1.85) | 48.19 (+2.48) |


For token placement accuracy, the results show that the overall accuracy is not bad except on the RefCOCOg dataset, which mostly consists of long expressions. Therefore, it is harder for our model to correctly identify the whole expression. We further analyze the placement accuracy for *\<obj\>* and *\</obj\>* separately, and we can find that our model can generate *\<obj\>* in the right place in most cases, while the generation of *\</obj\>* is not always correct. We empirically find that our model often fails in cases such as "an adult giraffe scratching its back with its horn". It will be parsed as "*<obj>*an adult giraffe scratching its back*</obj>* with its horn" or "*<obj>*an adult giraffe*</obj>* scratching its back with its horn". 

From the results of localization accuracy, we can see that the accuracy of using predicted token placement and the accuracy of using ground truth token placement is close. It indicates that our model is robust to inaccurate or ambiguous token placement.

Reviewer comment (Evaluation):
My primary concern is about the pre-training data and how to interpret the results. The pre-training dataset consists of text-image pairs from COCO, CC3M, CC12M, Visual Genome (Krishna et al., 2017), SBU, and LAION400M. The evaluation datasets share considerable overlap with the pre-training data. For example, COLA is composed of data from GQA (derived from Visual Genome), CLEVR, and LVIS (also includes COCO). Similarly, ARO is based on Visual Genome, and HICO-DET is based on Flickr (with objects limited to those from COCO). I understand that the training data is essentially the same as BLIP-2 and that it is common for VLMs to be trained on these datasets. The question does remain: given this overlap, how well does the model generalize? The generalization can be quantified based on two factors - tasks and domains. From the task perspective, the generalization of the proposed approach seems to be limited, going by the performance of VQA. There are no convincing arguments for the generalization beyond training data. How about the performance on PACO[1]? It does not have a **significant** overlap with the pre-training datasets and should provide some measure of generalization.

Author response:
> **Q1. Generalization capabilities.**

We thank the reviewer for recommending a related benchmark PACO [a] that is more suitable to provide some measure of generalization. We evaluate the AR@1 metric on all L1 queries under the zero-shot instance detection setting.

| Model | Type | PACO L1 AR@1 |
|---|---|---|
| MDETR R101 [b] | Open-vocabulary detector | 4.9 |
| Detic Swin-B [c] | Open-vocabulary detector | 5.9 |
| KOSMOS-2 | VLM | 8.2 |
| Compositional VLM 1.4B | VLM | 9.4 |

The results show that our method can generalize well to other image domains and harder tasks. Please see G3 for a detailed explanation of the results.

Reviewer comment (Evaluation):
There are very few qualitative results presented. There are 4 examples in the supplementary, but they are very limited beyond that. By reading the paper, it is hard to understand when/where/why the model fails. For example, is there a reason why the model has a higher performance on the "rare" class from HICO-DET? It would be good to understand the success and failure modes and have a discussion on the approach's qualitative performance as opposed to the purely quantitative take.

Author response:
> **Q2. There are very few qualitative results presented..**

Thanks. We have added more qualitative results and discussion in Appendix A.5 in our revision.

Reviewer comment (Experiments):
There are no ablations presented. What does the choice of encoders/pre-training data/tokens have on the model?

Author response:
> **Q3. There are no ablations presented. What does the choice of encoders/pre-training data/tokens have on the model?**

Thanks for your advice. We have added an ablation study on communication tokens in G1.


| Setting | <previsual> | <prebox> | <visual> | <box> | ARO Top-1 | Cola |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| 1 | x | x | x | x | 29.60 | 38.10 |
| 2 | x | x | ✓ | ✓ | 30.08 | 34.76 |
| 3 | ✓ | x | ✓ | x | 32.26 | 36.19 |
| 4 | ✓ | ✓ | ✓ | ✓ | **32.46** | **44.29** |

The ablation study results demonstrate the necessity and effectiveness of the proposed communication tokens. Please refer to G1 for detailed explanation.

For the choice of encoder, we follow previous works such as BLIP-2 [d] and LLaVA [e] to use pre-trained CLIP ViT-L [f] as our vision encoder, as the pre-trained CLIP ViT is proved to be a powerful model in various downstream tasks. For the choice of pre-training data, the only concurrent public large-scale grounded image-text dataset is from KOSMOS-2, but KOSMOS-2 only releases a 20 million subset of its full 90 million data. As the data quantity is important for multimodal pre-training, we choose to create our own large-scale grounded image-text dataset based on the widely used BLIP-2 pre-training data. Our pre-training dataset consists of over 97 million image-text pairs, comparable to KOSMOS-2's full pre-training data.

Due to the short period for rebuttal and the time-consuming nature of multimodal pre-training, it is hard for us to conduct an ablation study on encoders and pre-training data during the rebuttal period. We leave the additional ablation study on different encoders and different sets of pre-training data in our future work and we are happy to have more discussion with the Reviewer if needed.

Reviewer comment (Experiments):
On that note, how sensitive is the model to the template used for prompting?

Author response:
> **Q4. How sensitive is the model to the template used for prompting?**

Our model is robust to the template used for prompting. We evaluate the performance of using different prompts in RefCOCOg/RefCOCO+/RefCOCO benchmarks.

| Prompt | RefCOCOg val | RefCOCO+ val | RefCOCO val |
|---|:---:|:---:|:---:|
| Locate the object: <obj>{text}</obj><visual> | 60.64 | 46.83 | 47.50 |
| Locate: <obj>{text}</obj><visual> | 60.89 | 46.97 | 47.63 |
| Find: <obj>{text}</obj><visual> | 60.89 | 47.20 | 47.60 |
| <obj>{text}</obj><visual> | 60.87 | 47.62 | 48.19 |

The evaluation results do not differ much when using different prompts, indicating that our model is robust to the prompt template.

Reviewer comment (Presentation):
The presentation of the method is brief, making it hard to understand. It is not clear how the communication tokens are generated. I recommend providing a pseudo algorithm to describe the method and making the input/output for each step more readable.

Author response:
> **Q5. The presentation of the method is brief, making it hard to understand. It is not clear how the communication tokens are generated. I recommend providing a pseudo algorithm to describe the method and making the input/output for each step more readable.**

Communicate tokens are automatically generated during the auto-regressive generation process. We have added an explanation of the communication token generation in G1 and a pseudo algorithm in Appendix A.1 in our revision.

Reviewer comment (Novelty):
The proposed pre-training procedure seems to incorporate a lot of engineering tricks and data processing techniques from KOSMOS-2, without adequate attribution to previous works. The paper also lacks justifications for the selection of sub-components in the proposed method, making it hard to interpret the results.

Author response:
> **Q6. No adequate attribution to previous works.**

Thanks for your reminder. During our data-creating process, we have borrowed some techniques from KOSMOS-2 including using spaCy to parse the text and use *\<obj\>\</obj\>* to enclose an object. We also have some differences from KOSMOS-2. For example, we use GroundingDINO to directly ground the textual description in the text to the object in the image. In contrast, KOSMOS-2 first parses the sentence to extract all noun chunks. Then, they eliminate certain abstract noun chunks that are hard to recognize such as "time" and "love". Finally, they use GLIP to obtain the bounding boxes for these noun chunks. In comparison, our data-creating method is more simple but still effective. We have revised to give more adequate attribution to previous works in our revision.

> **Q7. Lack of justifications for the selection of sub-components in the proposed method.**
    
Thanks for your reminder. We have added the ablation study on communication tokens. Detailed results and analysis are shown in G2, which indicates the effectiveness and necessity of these tokens.

Reviewer comment (Evaluation):
Pretrained data and downstream tasks (i.e., RefCOCO/RefCOCO+, VQA v2) share the same pool of visual data taken from COCO and Visual Genome. This raises concerns about the validity of the reported performance on these downstream tasks.

Author response:
> **Q8. Pretrained data and downstream tasks (i.e., RefCOCO/RefCOCO+, VQA v2) share the same pool of visual data taken from COCO and Visual Genome.**

It is worth noting that in KOSMOS-2's instruction tuning stage, LLaVA-Instruct [b] data is used to fine-tune the model, and the images in LLaVA-Instruct data are from COCO. Therefore, KOSMOS-2's training data and downstream tasks also share the same pool of visual data from COCO. The visual data overlap concern is not unique to our model.

To prove the generalization of our model, we further evaluate our model on PACO [a] as recommended by reviewer Aw71.

| Model | Type | PACO L1 AR@1 |
|---|---|---|
| MDETR R101 [b] | Open-vocabulary detector | 4.9 |
| Detic Swin-B [c] | Open-vocabulary detector | 5.9 |
| KOSMOS-2 | VLM | 8.2 |
| Compositional VLM 1.4B | VLM | 9.4 |

Our model continues to perform well in this dataset, indicating our model's good generalization to other image domains. The dataset description and result analysis for the PACO dataset can be found in G3.

> **Reference:**
    
[a] PACO: Parts and Attributes of Common Objects. CVPR 2023.
    
[b] MDETR - Modulated Detection for End-to-End Multi-Modal Understanding. ICCV 2021.
    
[c] Detecting Twenty-thousand Classes using Image-level Supervision. ECCV 2022.

Reviewer comment (Experiments):
Why did you opt to create your own dataset of similar scale for pretraining, rather than utilizing the data from KOSMOS-2?

Author response:
> **Q1. Why do you create your own dataset rather than using data from KOSMOS-2.**

KOSMOS-2 has made their data public, but they have only released a **subset** of it, about 20 million image-text pairs instead of the full dataset with 90 million image-text pairs. A large-scale and high-quality dataset is essential for pre-training, therefore, we turned to create our own dataset which has a similar scale compared to KOSMOS-2. The dataset we create consists of over 97 million image-text pairs. We will make our full dataset public in the future.

Reviewer comment (Experiments):
Could you provide further analyses of the contributions of these communication tokens?

Author response:
> **Q2. Could you provide further analyses of the contributions of the communication tokens?.**

Communication tokens are designed to enable bidirectional communication between the vision module and the language module. They can summarize the compositionality information within text to guide the vision module to detect the desired objects, and can also bring the visual feature of the object back to the language module to guide the LLM to notice the localized visual feature, thus helping it to better generate more faithful and related tokens thereafter. We have added an ablation study on communication tokens in G1.

| Setting | <previsual> | <prebox> | <visual> | <box> | ARO Top-1 | Cola |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| 1 | x | x | x | x | 29.60 | 38.10 |
| 2 | x | x | ✓ | ✓ | 30.08 | 34.76 |
| 3 | ✓ | x | ✓ | x | 32.26 | 36.19 |
| 4 | ✓ | ✓ | ✓ | ✓ | **32.46** | **44.29** |

The ablation study results demonstrate the necessity and effectiveness of the proposed communication tokens. Please refer to G1 for detailed explanation.

Reviewer comment (Evaluation):
On the task REFERRING EXPRESSION COMPREHENSION: the proposed model outperforms the KOSMOS-2 model, but with a small gap. While the grounding of vision-language in KOSMOS-2 was reasonably good compared to the proposed model, it exhibited significantly poorer performance in the previous three tasks. Can you provide insights into these differences?

Author response:
> **Q3. The proposed model outperforms the KOSMOS-2 model in referring expression comprehension tasks but with a small gap. While the grounding of vision-language in KOSMOS-2 was reasonably good compared to the proposed model, it exhibited significantly poorer performance in the previous three tasks. Can you provide insights into these differences?**

The performance gap is small in referring expression comprehension tasks because KOSMOS-2 and our model both adopt a similar manner for testing. This gap is large in the previous three tasks because the communication tokens in our method can facilitate our model to understand object relations and focus on a localized ROI to generate more faithful object predictions. The deeper analysis is as below:

1. For **referring expression comprehension** tasks, both of us use a format of *\<obj\>*{text}*\</obj\>* as the prompt to the model, and then enable the model to output location information. KOSMOS-2 outputs some location tokens to represent the location, while our model outputs the bounding box through the object detector. The advantages of communicative decoding and the bidirectional communication between the vision module and the language module are not fully exploited in this task, thus the performance gap between KOSMOS-2 and ours is reasonably small.

2. The **HICO-DET** benchmark asks the model to detect all human-object interactions in the image. Many images contain more than one human-object interaction, that is, there are multiple objects corresponding to the same textual description that need to be identified correctly. We empirically find that compared to KOSMOS-2, our model will make it easier to detect more correct objects corresponding to the same correct textual description in the image. Therefore, our model can find out the more complete human-object interactions within an image. This is thanks to our YOLO-like object detector that can naturally identify all correct objects in the image. The input image will be encoded by the vision encoder to become a map of 16x16 patches. When performing the object detection, each patch will be concatenated with the hidden state of *\<previsual\>*/*\<visual\>* from LLM, and then fed to the object detector to get a score indicating the probability that this patch contains the center of the object, and also get a regressed bounding box coordinates indicating the left-top and right-bottom coordinates of the object if exists. Each patch can produce a probability and object bounding box individually, thus it is natural and convenient for our model to identify multiple objects.

3. **ARO** requires that given entity A and relation, the model should predict entity B to which the relation is applied. It requires the model to understand where is entity A and what's the relation and use this information to infer the entity that A is applying the relation to. The design of *\<previsual\>*/*\<prebox\>* communication tokens can handle this relation information well. By using the information of entity A and the relation, *\<previsual\>*/*\<prebox\>* communication tokens can guide the LLM to focus on the potential regions for entity B, so it would be much easier to correctly predict entity B. **Cola** benchmark also require the model to understand both the objects and their relation, thus *\<previsual\>*/*\<prebox\>* communication tokens can help in this task. Since KOSMOS-2 lacks communication tokens and explicit guidance for LLM to focus on a specific region, it does not perform well in such tasks.

Reviewer comment (Evaluation):
Can you clarify the large margins between KOSMOS-2 and the proposed method in Table 1? What would be the main contributor to this significant performance leap?

Author response:
> **Q4. Can you clarify the large margins between KOSMOS-2 and the proposed method in Table 1? What would be the main contributor to this significant performance leap?**

The large margins between KOSMOS-2 and our proposed method in Table 1 are mainly because our model has adopted a novel bidirectional communication between the vision module and language model via communication tokens, which brings a large improvement in compositional reasoning ability. Our ablation study especially shows the importance of the proposed communication tokens for compositional tasks (please refer to G2). KOSMOS-2 does not have this kind of design and thus performs much poorer in these tasks.