Paper: Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond (ICLR-2024)

Reviewer comment (Writing):
The claim in the related work section, "Despite achieving significant progress, previous vision-language models still have several limitations such as poor robustness in instruction following, limited generalization capabilities in unseen tasks, and a lack of in-context abilities." lacks justification. For example many models are not instruction-tuned (yet). That does not mean they have a fundamental difficulty in instruction following.

Author response:
**[Related work clarity]**

This is a missunderstanding, the limitations we mentioned were not intended to challenge previous vision-language models for lacking instruction tuning process. On the other hand, we meant to pinpoint the potential improving direction compared to existing vision-language chatbots (*e.g.*, VisualGLM, PandaGPT, MiniGPT4, and etc.). And as the results shown in Table 7, compared to these predecessors, Qwen-VL exhibits better instruction following ability on three recently proposed instruction following evaluation benchmarks.

*For better readability, we have updated this section in the revision.*

Reviewer comment (Experiments):
The ablation study is not written clearly (also see questions below).

Author response:
**[Ablation study details]**

Thanks for pointing this out! The experiments in Figure $7$ are all conducted in the **fist** stage, with $224\times224$ resolution for image inputs. That is to say, our vision encoder will pathify each input image into $256$ patches, and under this circumstance we found that inadequate and superfluous numbers of queries both lead to slightly higher loss in the later training period.

*We have updated this part in the revision to make it more clear.*

Reviewer comment (Experiments):
In the ablation study of Figure 7, which stage is that, lower-res or higher-res? If it is lower-res (224) stage it makes sense to use 256 tokens as the native number of patches is just (224/14)^2 = 256. If it is the higher-res (448) stage, then it is counterintuitive that using more tokens, i.e., 400, with more degrees of freedom, will lead to worse performance.

Author response:
**[Ablation study details]**

Thanks for pointing this out! The experiments in Figure $7$ are all conducted in the **fist** stage, with $224\times224$ resolution for image inputs. That is to say, our vision encoder will pathify each input image into $256$ patches, and under this circumstance we found that inadequate and superfluous numbers of queries both lead to slightly higher loss in the later training period.

*We have updated this part in the revision to make it more clear.*

Reviewer comment (Experiments):
seems missing generalist PaLI results. The PaLI-X authors also have multitask finetuned model for VQA and captioning mixtures separately.

Author response:
**[Miss PaLI results]**

Thanks for pointing this our. However, PaLI-X's multi-task finetuning remains treating different tasks (*e.g.*, image caption, VQA and etc.) seperately instead of as a whole (*i.e.*, one model to inference on all tasks). We are also glad to add more PaLI-X's multi-task finetuning results as specialist SOTAs for comparision.

Reviewer comment (Experiments):
missing the design / motivation or ablation of which part being trained during different stage. The stage 2 unfreezes ViT is for adapting to higher solution?

Author response:
**[Motivation and ablation on the training part in different training stages]**

First of all, we thank the reviewer for suggesting this ablation on the training part in different training stages. The motivation and intention of our choice on which part being tuned in different stage are listed as following:

* For the first stage, where billion-scale image-text pairs are leveraged, we tune the vision part only due to: (i) Efficiency: Tuning solely the vision component is much more efficient than joint tuning both ViT and LLM. (ii) Potential for catastrophic forgetting: There exists a significant likelihood of the LLM suffering from catastrophic forgetting if we simultaneously fine-tune ViT and LLM without access to a dedicated pure-text corpus. Notebaly, compared to the text corpus used for LLM pre-training, text lengths in image-text pair dataset (*e.g.*, CC3M, CC12M, Laion, and etc.) are much more noisy and short. To overcome this catastrophic forgetting issue, in the second and third stages, we incorporate pure-text corpus when LLM is tuned.
* For the second stage, the reasons behind jointly tuning ViT and LLM are two folds: (i) Additional Vision Tasks: We incorporate some additional vision tasks (*e.g.*, grounding and OCR) in this stage and all these tasks require fine-grained visual perception ability. To this end, we increase the input image resolution from $224$ to $448$ and tune ViT to adapt to this high resolution. (ii) Specific Output Requirements: Visual grounding asks the LLM to generate output in a specific behavior (*i.e.*, bounding box). Therefore, we choose to fune LLM to make this task doable.
* In the third stage, the focus shifts primarily towards supervised fine-tuning for dialogue capacity, rather than introducing additional visual capacities (which have been incorporated into ViT during the previous two stages). Consequently, we opt to freeze the ViT component while keeping the remaining parts trainable.

As per your request, we conduct additional ablation studies on the training part during the second stage. Given the limited amount of time and resources available to us in the response period, we are only able to get these experiments done in a small scale. Specifically, we employe a LLM with $1.8$B parameters and CLIP-ViT-L/$14$ as vision encoder, the overall parameters in our model are about $2$B. We train this model on the same dataset composition as we used in our paper, in spite of much fewer iterations. During the first stage, we tune the model with a batch size of $30720$ for $8000$ steps. During the second stage, in addition to the default setup, we also try to train our model with either ViT or LLM being frozen. All these three models for the second stage are tuned for $6000$ steps with a batch size of $2048$.

| Benchmark | Qwen-VL($2$B) | Freeze ViT in Stage$2$ | Freeze LLM in Stage$2$ |
|:---|:---:|:---:|:---:|
| Nocaps (CIDEr) | $114.7$ | $109.7$ | $99.3$ |
| VQAv2-val (VQA Score) | $74.6$ | $68.9$ | $64.4$ |
| OKVQA-val (VQA Score) | $45.7$ | $43.2$ | $32.7$ |
| VizWiz-val (VQA Score) | $29.4$ | $22.5$ | $19.3$ |
| TextVQA-val (VQA Score) | $52.5$ | $27.7$ |  $40.1$ |

There are several observations:

* Compared to jointly tuning both ViT and LLM, freeze ViT or freeze LLM both lead to significant performance decrease.
* On most benchmarks, freeze ViT shows better results than freeze LLM. This observation is in line with some current vision-language machines who prefer to tune LLM and keep the ViT frozen.
* However, for text-oriented VQA(i.e, TextVQA), it's interesting to see that freeze LLM (tune ViT) shows significant higher accuracy compared to freeze ViT (tune LLM). We ascribe this phenomenon to text-oriented VQA requires more fine-grained visual perception capacity which has not been learnt in previous ViT training process(*e.g.*, vision-language contrastive learning in CLIP, and coarse-grained image captioning in the 1st stage).

Reviewer comment (Novelty):
W1: Weak research significance and contributions. This work is a huge engineering effort and it is appreciated. However, research-wise, I am not convinced that it can provide any insights in terms of large-scale model training, architecture, or evaluation.

Author response:
**[Research contributions and insights]**

We thank the reviewer for valuable feedback. In addition to the contributions listed in our paper, we restate Qwen-VL's contribution, technical insights, and novelty point by point in the general response. We hope that our detailed reply will ease your concerns.

Reviewer comment (Novelty):
W2: Weak discussion of related work and clarity: To make W1 worse, the paper does not properly discuss the relevant work. If the paper would like to focus on the open-source aspect, I think it can expand this part much more heavily. What are the existing open-source LVLMs and what are “open” about them? What are the capabilities they support and so on? However, based on the current presentation this is unclear.

Author response:
**[Discussions of open-source work]**

We thank your kind suggestion. In the below table, we compare Qwen-VL with previous open-source large vision-language models (LVLMs) in terms of their supporting language and capable task. As shown, Qwen-VL can support both English and Chinese and is capable to finish four kinds of vision-language tasks within the same model. Moreover, as mentioned in our paper, Qwen-VL also outperforms these predecessors across several benchmarks on these four tasks. Based on the versatile ability of Qwen-VL and its superior performance on a wide range of benchmarks, we believe that Qwen-VL can serve as a strong foundation for both future research and application purpose.

*We trust our response can make the capabilities of previous LVLMs much more clear. We also include this discussion in our revised paper.*

> Note: For text-oriented VQA, we refer to whether the model is designed or optimized to tackle this problem explicitly.

| Model          | English | Chinese | Caption | General VQA | Text-oriented VQA | Grounding |
|----------------|:---------:|:---------:|:---------:|:-----:|:------------:|:-----------:|
| Kosmos         | $\checkmark$ |         | $\checkmark$ | $\checkmark$ |            |           |
| BLIP2          | $\checkmark$ |         | $\checkmark$ | $\checkmark$ |            |           |
| LLaVA          | $\checkmark$ |         | $\checkmark$ | $\checkmark$ |            |           |
| MiniGPT-4      | $\checkmark$ |         | $\checkmark$ | $\checkmark$ |            |           |
| ChatGLM        | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |            |           |
| mPLUG-Owl      | $\checkmark$ |         | $\checkmark$ | $\checkmark$ |            |           |
| InstructBLIP   | $\checkmark$ |         | $\checkmark$ | $\checkmark$ | $\checkmark$ |           |
| mPLUG-DocOwl   | $\checkmark$ |         | $\checkmark$ | $\checkmark$ | $\checkmark$ |           |
| Kosmos2        | $\checkmark$ |         | $\checkmark$ | $\checkmark$ |            | $\checkmark$ |
| Shikra         | $\checkmark$ |         | $\checkmark$ | $\checkmark$ |            | $\checkmark$ |
| Qwen-VL        | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |

Reviewer comment (Experiments):
Is the train-test overlap between benchmarks taken care of? Especially COCO-based datasets.

Author response:
**[Train-test overlap]**

We appreciate the reviewer for highlighting the potential image contamination between our training dataset and various evaluation benchmarks, notably the coco-based benchmarks. To assess and address this concern, we conducted two contrast experiments. However, due to constraints in both time and resources within the response period, these experiments were performed on a smaller scale. Below, we outline the setups of our experiments:

* We use a LLM with $1.8$B parameters and CLIP-ViT-L/$14$ as vision encoder. Total parameters are about $2$B.
* We perform the first and second training stages, and evaluate the models on several caption/VQA/Grounding benchmarks.
* For the first stage, we train the model with a batch size of $30720$ for $8000$ iterations. In other words, the total consumed image-text pairs are about $245.76$M.
* For the second stage, we train our model with a batch size of $2048$ for $6000$ steps.

For the baseline experiment, we use exact the same composition of datasets as in our paper. For the second experiment, we follow previous work to perform near-deduplication. Specifically, we deduplicate all coco images used in some coco-based evaluation benchmarks (*e.g.*, VQA, OKVQA, Refcoco/Refcoco+/Refcocog). The final results are shown in table below:

| Benchmark | Is COCO-based? | Qwen-VL($2$B) | Qwen-VL($2$B) + Data Deduplication |
|:---|:---:|:---:|:---:|
| Nocaps (CIDEr) | | $114.7$ | $114.3$ |
| VQAv2-val (VQA Score) | $\checkmark$ | $74.6$ | $74.3$ |
| OKVQA-val (VQA Score) | $\checkmark$ | $45.7$ | $45.9$ |
| VizWiz-val (VQA Score) | | $29.4$ | $29.8$ |
| TextVQA-val (VQA Score) | | $52.5$ | $52.5$ |
| RefCOCO-val (Accuracy) | $\checkmark$ | $79.9$ | $79.6$ |
| RefCOCO+-val (Accuracy) | $\checkmark$ | $66.1$ | $66.5$ |
| RefCOCOg-val (Accuracy) | $\checkmark$ | $74.2$ | $73.6$ |

From the results, we observe little to no performance perturbation between two experiments for both coco-based and non coco-based benchmarks, which demonstrates the reliability of our results. We also appreciate the reviewer's professional comment.