Paper: CogVLM: Visual Expert for Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
Comparing the proposed method to earlier approaches such as PaLI, CoCa, and Flamingo may not be entirely fair. These prior methods do not incorporate the SFT stage, making it unclear how the model performs before this crucial phase.

Author response:
This is a misunderstanding. We did not incorporate SFT to evaluate the benchmarks, so the evaluation is fair. See 2. of "response to common concerns" for details.

Reviewer comment (Experiments):
Since the pretrained image encoder and LM went through VLM finetuning, their original behavior may have changed. I wonder what the visual evaluation (linear probe, zero-shot) will be for this finetuned encoder, compared to the original model.

Author response:
This is also a misunderstanding. During the pretraining, the parameters of both the ViT image encoder and the LLM were completely **frozen**, thus the ViT is not affected, because it is also frozen in SFT.  LM is trainable during SFT with a very small learning rate of 1e-6 for 8,000 iterations, which might affect the language style but not much on knowledge.

Reviewer comment (Experiments):
How was LM performance affected by the VLM finetuning?

Author response:
This is also a misunderstanding. During the pretraining, the parameters of both the ViT image encoder and the LLM were completely **frozen**, thus the ViT is not affected, because it is also frozen in SFT.  LM is trainable during SFT with a very small learning rate of 1e-6 for 8,000 iterations, which might affect the language style but not much on knowledge.

Reviewer comment (Novelty):
The idea is not that novel, compared with BEIT-3 and VLMo, which also introduces different modality expert structures, although this work makes some changes to make it work in the era of LLM.

Author response:
Our response to this concern is in  1. of "reply to common concerns" above. The MoE is definitely not a new idea, but in the current LLM+ paradigm, it is very valuable to rediscover its effectiveness as a way to better scaling up.

Reviewer comment (Experiments):
The VQA/Image Caption model, Visual Grounding model and Chat model are three different models, I am wondering how the performance can be if all these models are a unified one? Since GPT-4V may be a unified one.

Author response:
Previously, we did not unify the models for VQA, chat, and grounding tasks due to their diverse output requirements, such as single-word answers for VQA, lengthy analytical responses for chat, and coordinate-heavy answers for grounding. However, our latest experiments indicate that VQA and chat models can be effectively unified. We will soon release a unified checkpoint where simple prompt modifications like "Short answer:" and "Answer:" enable the model to perform strongly in both VQA and chat domains, as shown in our latest results in Table 1(The touchstone score increase from 663 to 740). 

The grounding task is slightly different; our results show it slightly reduces the model’s captioning ability. For instance, performance on the COCO dataset dropped by 2 points and on the NoCaps dataset by 1.5 points, potentially due to the model’s limited understanding of coordinates. Conversely, grounding training improved the model's performance by 0.8 point on the  VQAv2 dataset, likely due to enhanced object localization capabilities. We believe a fully unified model is achievable and are working towards this, with adaptations needed for grounding tasks, such as adding specific coordinate-representing words to the vocabulary and loosening parameters in the word embedding layer and LM head during training.

Reviewer comment (Experiments):
Although I appreciate the excellent performance it achieves, the visual backbone is ViT-e, and the input resolution is 490 * 490, also the parameter size of LLM doubles, which makes the comparason a little hard.

Author response:
Our comparison is fair enough actually. It's common for multimodal models to upscale resolution for downstream tasks to enhance performance, similar to BLIP-2 (490 * 490) and Qwen-VL (448 * 448), with Google’s PaLI-X even using 896 * 896 resolution for image inputs.
For ViT-E, we also experimented with the OpenCLIP-L visual encoder, comprising 300M parameters, and found its performance only slightly worse than that of ViT-E. For instance, the score on the VQAv2 test set was 83.4 with OpenCLIP-L compared to 84.7 with ViT-E, and 88.6 on the Visual-7W dataset versus 90.6 with ViT-E.

Reviewer comment (Experiments):
Do you have more experiment on the archtecture of visual expert module and more insight about which part of the layers should be shared and which module should have separate parameters?

Author response:
In the ablation study part, we experimented this. The finding basically shows that the best way is to insert new parameters in each layer.

Reviewer comment (Experiments):
For the generalist performance, is it possible that a model can achieve best performance on both real-world chat and benchmark datasets? since this paper has three separate training procedures to make it best in each individual dataset. If there exist some gaps between different kinds of datasets, how can the architecture be designed to better address this problem?

Author response:
Yes, we merged the training of chat and generalist on benchmarks, which is reported in the response to the second question. The results are even better on chat and basically the same for benchmarks (generalist).

Reviewer comment (Evaluation):
Have you observed some new emergent ability in this strong MLLM?

Author response:
The definition of emergent abilities in the multimodal domain is not entirely clear. However, in our recent evaluations, we have indeed observed that our model shows significantly greater improvements in challenging tasks compared to other models, beyond what we see in VQAv2. For example, in the OCR&Spat task defined in MM-Vet, our model scored 59 compared to 14 for InstructBLIP, and in the Rec&Know task, our score was 39 while MiniGPT-4 scored 0.

Reviewer comment (Experiments):
The perhaps biggest weakness with this paper is the writing. This paper starts by raising two possible drawbacks of shallow alignment methods: (i) converge fast but perform worse, and (ii) weak visual understanding ability, especially hallucination. However, both these two disadvantages proposed by the authors are just **hypothesises**, not **compelling** nor **conclusive**. First, the performance gap between BLIP-2 and PaLI-X comes from several possible differences between the two frameworks (e.g., the visual encoder size, the way that the visual encoder is pre-trained). Furthermore, both MiniGPT-4 and LLAVA have extremely few trainable parameters in the alignment between visual features and language features.

Author response:
> 1. However, both these two disadvantages proposed by the authors are just hypothesises, not compelling nor conclusive. First, the performance gap between BLIP-2 and PaLI-X cames from several possible differences between two framework (e.g., the visual encoder size, the way that visual encoder is pre-trained by). And both MiniGPT-4 and LLAVA have extremely little trainable parameters in the alignment between visual features and language features.

This point is about the effectiveness of deep fusion. The comparison between PaLI-X and BLIP-2 are  just two examples of shallow / deep alignment methods, and not a strict claim. You are right that they cannot be compared directly. We will modify it in the final version to avoid confusion.

Reviewer comment (Evaluation):
Some blanket statements are used. For instance, the author claims that NLP ability is weakened when jointly training the language model in image-text training. However, there is some evidence showing that jointly training can benefit both the vision task as well as the language task, at least in some aspects (e.g., [1]).

Author response:
> 2. Some blanket statements are used. For instance, the author claims that NLP ability is weakened when jointly train the language model in image-text training. However, there are some evidences show that jointly training can benefit both vision task as well as language task, at least in some aspect (e.g., [1]).

It is not a blanket statement. In our paper, the next sentence of this statement is "According to PaLM-E [2]...", which runs experiments to prove the catastrophic forgetting in LLMs during jointly training. Specifically, Palm-E without freezing the language model parameters during multimodal training, showed a 15% drop in NLU capability and an 87% drop in NLG capability across 21 NLP tasks.  Similar findings were observed in Flamingo, where not freezing the LLM resulted in an average 8% performance decrease across five multimodal benchmarks. Regarding the 'Sight Beyond Text' paper, it only trained with a smaller learning rate on approximately 600,000 data points, which likely led to minimal changes in the LLM parameters, potentially resulting in a significant gap from large-scale pretraining. Hence, it may not be a proper reference to prove no catastrophic forgetting in LLMs during jointly training. For hallucination problems, our model has a significantly higher F1 score on the challenging POPE dataset than other models, which confirms the statement in our paper. See "Results on hard benchmarks" for details.

[2] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multi-modal language model. arXiv preprint arXiv:2303.03378, 2023.

Reviewer comment (Experiments):
The motivations and starting points are inconsistent with the experiments. In other words, despite the strong performance, the ablation studies cannot demonstrate that the two problems of shallow alignment raised by the writers are well resolved. The ablation studies in Table 6 can prove the effectiveness of CogVLM design, but these numbers cannot prove that deep alignment is better than and solves the issues of shallow alignment, due to the results of the shallow alignment method with a larger visual encoder (same parameters as vision encoder + vision adapter) remaining unknown.

Author response:
> 3. Ablation studies cannot prove that deep alignment is better than and solves the issues of shallow alignment, due to the results of shallow alignment method with larger visual encoder (same parameters as vision encoder + vision adapter) are remain unknown.
This proposed ablation study is very reasonable but unfortunately is very hard to implement, because the ViT (CLIP) is pretrained and the performance is not solely determined by the number of parameters. We cannot get a good ViT with arbitrary number of parameters easily for comparison.

However, we can still use a hypothetical comparison. In our paper, the  "VE-full every 4th layer" setting (1.7B) performs much better that the "MLP-adapter"(shallow) setting (140M), (COCO CIDEr 131.2 vs 138.7). If there were a well-trained EVA-CLIP ViT of (5B+1.7B) parameters, it is not possible to increase that much in "MLP-adapter" setting, since the current "MLP-adapter" has already 5B parameters.

If you make the ViT trainable to increase the trainable parameters, the performance will not be better either, which is well-documented in previous works like flamingo[3] (Table 7 (xii) 68.4 ->64.5 after making vision encoder trainable).
I hope this hypothetical comparsion can ease your concern.

[3] Alayrac, Jean-Baptiste, et al. "Flamingo: a visual language model for few-shot learning." Advances in Neural Information Processing Systems 35 (2022): 23716-23736.

Reviewer comment (Experiments):
Section 2.2 mentions that CogVLM is trained via a two-stage process, with 120K and 60K steps respectively. The ablation studies in Table 6 are trained for just 6K steps. However, despite having much fewer iterations, the performance gap between the ablation model and the final model is not that significant (e.g., in Table 6, CogVLM achieves 142.8 COCO CIDEr, only ~4 CIDEr score less than the results in Table 3). This raises the question of whether too many iterations in the two-stage training process are unnecessary.

Author response:
> 4. despite with much fewer iterations, the performance gap between ablation model and the final model is not that significant (e.g., in Table 6, CogVLM achieves 142.8 COCO CIDEr, only ~4 CIDEr score less that the results in Table 3). So does this phenomenone implies that too much iterations in the two-stage training process are unnecessary?

More training iterations are important. Regarding dataset evaluations, the COCO dataset is relatively simple and ground truth is relatively fixed in style and mostly includes common everyday scenes.  Thus, continued training does not show significant improvement, indicating a need for more challenging benchmarks in the research community, like MM-Vet. We observed that our model's performance continued to grow with training, ultimately far surpassing other models. CogVLM achieved a new sota of 52.8 for MM-Vet, which will be updated in our paper. Please see our comment titled "Result for hard datasets" above.

Reviewer comment (Experiments):
The visual expert in CogVLM includes FFNs in both the attention block and the FFN block. It is unclear which one is more important for better performance.

Author response:
> 5. The visual expert in CogVLM includes FFNs in both attention block and FFN block. Which one is more important for better performance?

The ablation experiments in Table 6 suggest that the FFN plays a slightly more significant role than the attention layer. Adding a visual expert every four layers also yielded competitive results. Therefore, choosing a model structure will depend more on the balance between desired model performance and training efficiency.

Reviewer comment (Reproducibility):
In section 2.3, the author claims that errors in the LLAVA-Instruct dataset are corrected by manual inspection and annotation. It is unclear whether the corrected dataset will be made publicly available.

Author response:
> 6.  Will the corrected dataset be made publicly available?

Yes, we will release the dataset if you think this behavior can benefit the community.