Paper: Grounding Multimodal Large Language Models to the World (ICLR-2024)

Reviewer comment (Reproducibility):
While generally clear, I do not think I would be able to reproduce the pipeline of Figure 2. The submitted code does not seem to include the steps required to obtain data similar to what is in GrIT. This heavily limits the applicability of the proposed method and future comparisons that aim at improving data manipulation strategies.

Author response:
Yes, we will open-source the code to reproduce the pipeline of Figure 2 to produce dataset. We have updated the supplementary materials with the code of the pipeline. Please refer to it for more details.

Reviewer comment (Experiments):
Using Flickr30K as captioning benchmark is suboptimal due to the small number of images (1K) – I encourage the authors to report performance on COCO, which most work uses as benchmark for captioning.

Author response:
Thank you for your suggestions. We have evaluated the model on COCO captioning. The table below shows the zero-shot results on COCO captioning. Kosmos-2 also achieves competitive performance on COCO benchmark.

| Model | CIDEr |
| --- | --- |
| VLKD | 58.3 |
| MetaLM | 82.2 |
| Flamingo-9B | 79.4 |
| Kosmos-1 | 84.7 |
| Kosmos-2 | 90.0 |

Reviewer comment (Experiments):
While the authors claim that “grounding capability born with KOSMOS-2 enables it to be applied to more downstream tasks, such as grounded image captioning, and grounded visual question answering” no such tasks are tested. An evaluation on a dataset like “VizWiz-VQA-Grounding” [1] would be appreciated towards grounding these claims. Likewise, claims about better action modeling (Conclusion) are unrelated to the proposed model.

Author response:
We appreciate your suggestion to test Kosmos-2 on the VizWiz-VQA-Grounding dataset. We tried to evaluate the model on the task but find that VizWiz-VQA-Grounding dataset measures performance using mask IoU&mAP, while Kosmos-2 currently only supports bounding box output. We have reached out to the authors of the dataset and were informed that no bounding box metric has been prepared or tested. Given these circumstances, we instead measure the accuracy metric as in RefCOCO. When the bounding rectangle of the ground truth mask is used as the ground truth box, Kosmos-2 achieves 58.9% top-1 accuracy under the zero-shot setting.

Reviewer comment (Novelty):
Another concern that I have is about the unnecessarily claims towards world modeling and AGI. I agree that (better) grounding LLMs to vision goes towards these goals, however, I do not think this should be stressed in the title and abstract (which I invite the authors to consider modifying). While adding additional visual grounding abilities, this approach does not ground “models to the world” in a much more significantly extended way than concurrent work. This can mislead the readers, and given the increased interest in our field from the general public, I believe we researchers should not make claims stronger than needed. The paper proposes good contributions to improve visual grounding, and I think they should be framed as such.

Author response:
Thank you for your suggestions, we will try to correct those inappropriate claims

Reviewer comment (Novelty):
The Related Work section needs substantial expansion. Discussion on models like PEVL [2] and X-VLM [3] that pretrain VLMs with object detectors are missing, as well as some of their capabilities [4]. There is also no mention of other approaches towards data curation that can improve model performance, including work like BLIP [5], VSG [6] and other relevant work in NLP [eg, 7].

Author response:
We have added the discussion of these related works into Appendix F of the revised paper.

Reviewer comment (Evaluation):
The proposed model achieved much worse performance than the previous grounding and referring models, whereas ideally, with language model's stronger language capabilities, the model should achieve better performance on these tasks. Therefore, there is still room for improvement.

Author response:
We agree that there is a significant room for improvement. This can be achieved by increasing the model size and enhancing the image resolution. Regarding to the performance gap between our model and previous specialist models, we believe there are the following reasons:

1. Previous specialist models undergo pretraining or finetuning procedures on these specific datasets. In contrast, Kosmos-2 is trained on noisy web-scale data and is evaluated under zero or few-shot settings. This difference in training and testing conditions could partially explain the observed performance gap. We believe that increasing the model size can effectively narrow the gap in performance. We can also add the specific datasets, such as RefCOCO, into our training data to boost the performance. This has been acknowledged to yield good results by models such as QWen-VL[1], CogVLM[2], MiniGPT-v2[3] and Florence-2[4].
2. In order to reduce the cost of model training and inference, the input image resolution of our model is 224x224 and we reduce the number of image embeddings to 64 via a resampler before feeding them into the language model. While previous specialist models often employ a much larger image resolution to obtain better performance. We will increase the image resolution and number of image embeddings in future work.

Reviewer comment (Experiments):
There is curiosity about the performance of integrating an object-detection model with Kosmos-2. Specifically, if the location of the objects is provided for VQA tasks, what performance can the model achieve?

Author response:
We tested some VQA examples on the model and provided location information for relevant objects in the questions. We found that for some examples where the object being questioned is not clearly referred to by the text referring expression, providing their location information can help the model get the correct answer. Providing location information can resolve coreference ambiguity. In addition, we also tested the model on the grounded VQA benchmark VizWiz-VQA-Grounding, as suggested by Reviewer 873R, and achieved 58.9% accuracy. The model can locate the image region used to arrive at the answer.

Reviewer comment (Novelty):
Limited methodology innovation. There is no significant methodology improvement, as outputting bounding boxes as discrete tokens has been extensively studied in prior work such as [1]. [1]. OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework. Wang et al., 2022.

Author response:
While we acknowledge the contribution of Pix2Seq and other works such as OFA in considering bounding boxes as tokens, we believe our work, Kosmos-2, has its unique positioning and contributions:
1. Although both OFA and Kosmos-2 utilize the concept of bounding boxes as tokens from Pix2Seq, the motivations and goals of the two models are different. OFA aims to unify various cross-modal and mono-modal tasks via a unified sequence-to-sequence learning paradigm for pretraining and finetuning, and it has achieved impressive performances on a wide range of benchmarks. In contrast, Kosmos-2 focuses on unlocking the referring and grounding capabilities of MLLMs and integrates these capabilities into downstream applications in an open-ended style.
2. We are the first to verify the feasibility of using a simple yet effective 'hyperlink' modeling strategy to equip Multimodal Large Language Models with grounding capabilities while preserving conventional functionalities.
3. We also propose a straightforward pipeline for constructing web-scale grounded image-text pairs to enable potential applications for the community. We are delighted to see that our pipeline has been helpful to other works within the field.

Reviewer comment (Experiments):
While the evaluation is extensive, there is no ablation study or discussion of key design choices or error analysis. Thus it is hard to draw further insights from the experiments. The reviewer encourages the authors to include ablation study / error analysis.

Author response:
Thank you for your valuable suggestion on the inclusion of an ablation study. We have done some ablation studies and have included the results and analysis in the Appendix D of the revised paper. We summarize the results as follows:
| Ablation Settings | RefCOCO (val) | RefCOCO (testA) | RefCOCO (testB) | RefCOCO+ (val) | RefCOCO+ (testA) | RefCOCO+ (testB) | RefCOCOg (val) | RefCOCOg (test) | Flickr30k (val) | Flickr30k (test) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline | 52.3 | 57.4 | 47.3 | 45.5 | 50.7 | 42.2 | 60.6 | 61.7 | 77.8 | 78.7 |
| - Instruction tuning | 49.2 | 56.3 | 43.8 | 45.1 | 50.1 | 40.6 | 59.6 | 60.0 | 76.7  | 77.5 |
| - Step 2 in Figure 2 | 41.4  | 45.1  | 38.1  | 35.7  | 37.1  | 34.3  | 29.2  | 28.3 | 78.5  | 79.0 |

Reviewer comment (Experiments):
How much impact does the data cleaning process (Section 2) have on the final performance?

Author response:
Thank you for your valuable suggestion on the inclusion of an ablation study. We have done some ablation studies and have included the results and analysis in the Appendix D of the revised paper. We summarize the results as follows:
| Ablation Settings | RefCOCO (val) | RefCOCO (testA) | RefCOCO (testB) | RefCOCO+ (val) | RefCOCO+ (testA) | RefCOCO+ (testB) | RefCOCOg (val) | RefCOCOg (test) | Flickr30k (val) | Flickr30k (test) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline | 52.3 | 57.4 | 47.3 | 45.5 | 50.7 | 42.2 | 60.6 | 61.7 | 77.8 | 78.7 |
| - Instruction tuning | 49.2 | 56.3 | 43.8 | 45.1 | 50.1 | 40.6 | 59.6 | 60.0 | 76.7  | 77.5 |
| - Step 2 in Figure 2 | 41.4  | 45.1  | 38.1  | 35.7  | 37.1  | 34.3  | 29.2  | 28.3 | 78.5  | 79.0 |

Reviewer comment (Experiments):
Where does the in-context learning ability come from? Is it solely from interleaved image-text data or are in-context grounded data specifically included?

Author response:
We do not incorporate in-context grounded data, but we are very interested in trying in the future. Determining the precise source of the in-context learning ability is a complex task as it involves several factors such as data distribution, model architecture, and scale. This is indeed a fascinating and significant research topic in the age of LLM. In the context of Kosmos-2, the use of interleaved image-text data and text corpus seems to be related to the in-context learning ability on vision-language tasks.  We believe the few-shot capability is general and not specific to any datasets. We have deliberately avoided using previous annotated data to prevent the model from learning dataset-specific bias. Evaluating on grounding tasks such as Flickr30k should be feasible given the generality of the model's in-context learning ability.

Reviewer comment (Experiments):
Can we fine-tune Kosmos on downstream grounding tasks? If so, will the system match previous fine-tuned models?

Author response:
Absolutely, Kosmos-2 can be fine-tuned on downstream grounding tasks. However, when it is solely fine-tuned on existing grounding datasets, the language model of Kosmos-2 tends to degenerate into a task layer, similar to the Transformer in Pix2Seq[1]. This is not our intended goal. In fact, some subsequent works such as Qwen-VL[2] and CogVLM[3], which incorporate RefCOCO series datasets into the instruction tuning phase, have demonstrated that the final performance can match that of specialist models.

Reviewer comment (Experiments):
Few-shot in-context prompting results are shown only for RefCOCOg. Could we also do few-shot prompting for grounding tasks such as Flickr30K?

Author response:
We do not incorporate in-context grounded data, but we are very interested in trying in the future. Determining the precise source of the in-context learning ability is a complex task as it involves several factors such as data distribution, model architecture, and scale. This is indeed a fascinating and significant research topic in the age of LLM. In the context of Kosmos-2, the use of interleaved image-text data and text corpus seems to be related to the in-context learning ability on vision-language tasks.  We believe the few-shot capability is general and not specific to any datasets. We have deliberately avoided using previous annotated data to prevent the model from learning dataset-specific bias. Evaluating on grounding tasks such as Flickr30k should be feasible given the generality of the model's in-context learning ability.

Reviewer comment (Reproducibility):
Will some of the grounded data (converted from public data) be made public?

Author response:
Yes, we plan to make the grounded data, converted from public data, available to the public. We will also release all the associated code and model weights. This will facilitate further research in this field and enable others to reproduce and build upon our work.