Paper: What Makes for Good Visual Tokenizers for Large Language Models (ICLR-2024)

Reviewer comment (Writing):
The authors need to better define the important notions and consequently clarify the difference to previous works. Describing the difference between "overall semantics" and "fine-grained perception" is especially critical, as both terms seem vague to the readers but are the foundation of the evaluation benchmarks and insights in the paper.

Author response:
**Weaknesses 1.1 & Question 1.1 [semantics & fine-grained perception]**

We used “overall semantics” to represent “global understanding” of the image while using “fine-grained perception” to signify local or regional understanding of the image. Based on your suggestions, we have revised our explanation for this part. Please see below.

Reviewer comment (Writing):
The separation of object counting and multi-class identification from VQA is unclear. While the intuition of adding these tasks is understandable, clarification in Sec. 2.1 should be improved according to the reviewer’s related questions.

Author response:
**Weaknesses 1.2 & Question 1.2 [Comparing OC&MCI with VQA]**

We agree that there are questions asking "how many" in VQAv2. However, these questions are sometimes coupled with other semantic understanding. For example, VQAv2 may ask questions like "How many people are boarding a bus?". To answer this question, the model needs to understand semantic relations. In contrast, our OC and MCI questions are decoupled from such semantics, making them more focused on fine-grained visual understanding. 


To better clarify the difference in these tasks, we revised  Figure 1, Sec 2.1 and Table 1 in our revision, providing better distinguish and comparison among the four tasks. In particular, we revise the evaluation dimension of VQAv2 as "general semantics" due to its versatility to evaluate various visual understandings such as relations and actions. We take image captioning as the task to evaluate the overall semantics, which requires the model to understand the global information of the image. Then, OC and MCI are responsible for evaluating region-level and instance-level understanding, respectively.

Reviewer comment (Experiments):
In table 1 and 2, authors use CLIP for ablation study but switch to EVA-CLIP for the remaining experiments. Why is there a inconsistency?

Author response:
We use CLIP for empirical studies since it is a prime model and widely used in other tasks. 
To build a strong MLLM  for the other experiments, we resort to EVA-CLIP for a stronger model.

We provide the following results to show that EVA-CLIP is indeed stronger than CLIP in our GVTBench, which is aligned with the observations in [1]. 
|VQA|Caption|COCO-OC|COCO-MCI|VCR-OC|VCR-MCI|Avg|
|---|---|---|---|---|---|---|
|CLIP|47.7|64.2|45.5|**88.0**|34.5|68.8|58.1|
|EVA-CLIP|**48.1**|**65.3**|**46.0**|**88.0**|**34.6**|**69.1**|**58.5**|

Reviewer comment (Evaluation):
Based on table 5 and 6, it seems that there are consistent improvements by explicitly utilizing object-level annotations during pretraining. This does not necessarily mean that the proposed tokenizer is a good visual tokenizer, especially considering the performance drop on VQA and image captioning tasks.

Author response:
We agree that using different levels of annotations may lead to distinct final performances. For the comparisons in Table 5 and 6, we strictly used identical settings and show that FD indeed improves fine-grained visual understanding. From such fair comparisons, we safely conclude that the proposed GVT is a better visual tokenizer.

Reviewer comment (Experiments):
As mentioned earlier, I like the first part of this work studying the effectiveness of different visual tokenizers for LMMs. The argument of LMMs needing both semantic-rich and fine-grained representations is valid and shared by many but has never been demonstrated experimentally before. However, I do not see a clear motivation to propose a distillation method to address the problem derived from the observation at the first part. The authors simply distill the visual representations from EVA-CLIP to another randomly initialized vision encoder and presume that the new model has better semantic and fine-grained representations.

Author response:
* We observe from Table1-3 that CLIP is strong at semantic understanding (VQA and image captioning) while inferior to MAE at fine-grained understanding (OC and MCI); MAE is superior in fine-grained visual understanding compared to other self-training models, which can be attributed to its region/patch-level pretraining.

* Considering that MAE's ability of fine-grained visual understanding arises from its patch-level pretraining (which is realized by the patch reconstruction), we therefore seek to enhance CLIP with patch-level training. Since the mask-based patch-level training (EVA) is proven to be unsatisfactory (Table 3), distilling features matches our demands exactly because we can use both CLIP and patch-level distillation target.

Reviewer comment (Experiments):
I do not buy that the proposed method learns better fine-grained representations in that the input images for distillation are still 224x224, and the distillation is simply copying one network to another based on a limited number of images from ImageNet-1K. Despite the better performance in Table 4, I can see the unfairness when comparing the proposed method with previous ones -- e.g. different models have different settings and pretraining data.

Author response:
**[Why FD on imagenet]** The reason that we chose ImageNet as the dataset for distillation is due to its high quality of images. Furthermore, we found that the performance improvements are generalizable across various different datasets (See the point below).

**[More benchmarking results and generality of visual tokenizer]** 
We provide the evaluation comparison on the SEED-Bench [4], which is a recently released MLLM benchmark and built upon web-images in CC3M . The results show that FD can still improve the performance on the web image benchmark across a wider range of tasks, especially on fine-grained understanding tasks such as instance identity, location, and counting.

|			 |Scene Understanding|Instance Identity|Instance Location|Instance Attributes|Instance Counting|Spatial Relations|Instance Interactions|Visual Reasoning|Avg|
|---|---|---|---|---|---|---|---|---|---|
|GVT wo FD|41.26|34.30|31.40|**29.84**|34.81|**32.98**|**31.96**|50.75|35.93|
|GVT	      |**41.74**|**35.50**|**31.79**|29.45|**36.17**|31.96|**31.96**|**51.06**|**36.20**|

These results show that our visual tokenizer is effective at various vision-language tasks. On the other hand, the effectiveness of FD on various vision tasks has been shown in [3]. 


**Question 2 [Unfair comparison with Previous work]**

We agree that previous work (MiniGPT-4, LLaVa and BLIP-2) utilize different training protocols. However, it is difficult to complemently align all the settings due to the distinct differences among these models. Our results in Table 4 (Table 5 in ours revision) aims to provide a systematic comparison with existing models, so as to demonstrate our model is able to achieve superior performance. 
However, we would like to highlight that, in our empirical studies and ablation studies, the comparisons follow the same experimental setting. The fair comparison in these experiments enable us to draw useful conclusions and reliable insights on the training supervision of visual tokenizers.

Reviewer comment (Experiments):
In the ablation study, the authors did show some improvements after distillation on the new benchmarks. Two question marks arise. First, why the authors use EVA-CLIP which proved to be defective in Table 3. Second, is it because the distillation pulls the feature space closer to the target domain, e.g., web images->imagenet->COCO? In Table 6, the gap actually is very marginal (seems the averaged gap does not match the individual gaps for Flant5-xxl).

Author response:
We would like to clarify that EVA [1] and EVA-CLIP [2] are two different models. EVA is a Mask-Image-Modeling pretrained model which uses CLIP feature as region supervision, while EVA-CLIP is a contrastive vision-language pretrained model. In essence, EVA is more of an MAE model while EVA-CLIP is more of a CLIP model. Table 3 shows that **EVA** is less useful but we use **EVA-CLIP** as a stronger version of CLIP for feature distillation to achieve a strong MLLM. The comparison between CLIP and EVA-CLIP is shown as follows:

||VQA|Caption|COCO-OC|COCO-MCI|VCR-OC|VCR-MCI|Avg|
|---|---|---|---|---|---|---|---|
|CLIP|47.7|64.2|45.5|**88.0**|34.5|68.8|58.1|
|EVA-CLIP|**48.1**|**65.3**|**46.0**|**88.0**|**34.6**|**69.1**|**58.5**|

The results show that EVA-CLIP is generally better than CLIP in various tasks.

**[Why FD on imagenet]** The reason that we chose ImageNet as the dataset for distillation is due to its high quality of images. Furthermore, we found that the performance improvements are generalizable across various different datasets (See the point below).

Reviewer comment (Experiments):
The experimental study is not enough to demonstrate the effectiveness of the distilled visual tokenizer. The authors should examine the visual tokenizers in a wider range of vision and vision-language tasks. That being said, the authors attempt to find a more generic visual tokenizer that can provide semantic-rich and fine-grained representations but fail to demonstrate the generality of the proposed visual tokenizer by itself.

Author response:
**[More benchmarking results and generality of visual tokenizer]** 
We provide the evaluation comparison on the SEED-Bench [4], which is a recently released MLLM benchmark and built upon web-images in CC3M . The results show that FD can still improve the performance on the web image benchmark across a wider range of tasks, especially on fine-grained understanding tasks such as instance identity, location, and counting.

|			 |Scene Understanding|Instance Identity|Instance Location|Instance Attributes|Instance Counting|Spatial Relations|Instance Interactions|Visual Reasoning|Avg|
|---|---|---|---|---|---|---|---|---|---|
|GVT wo FD|41.26|34.30|31.40|**29.84**|34.81|**32.98**|**31.96**|50.75|35.93|
|GVT	      |**41.74**|**35.50**|**31.79**|29.45|**36.17**|31.96|**31.96**|**51.06**|**36.20**|

These results show that our visual tokenizer is effective at various vision-language tasks. On the other hand, the effectiveness of FD on various vision tasks has been shown in [3].

Reviewer comment (Theory):
I think the authors should explain a bit more about how to derive the proposed distillation methods to improve the visual tokenizer and why the new tokenizer is supposed to learn more fine-grained representations.

Author response:
* We observe from Table1-3 that CLIP is strong at semantic understanding (VQA and image captioning) while inferior to MAE at fine-grained understanding (OC and MCI); MAE is superior in fine-grained visual understanding compared to other self-training models, which can be attributed to its region/patch-level pretraining.

* Considering that MAE's ability of fine-grained visual understanding arises from its patch-level pretraining (which is realized by the patch reconstruction), we therefore seek to enhance CLIP with patch-level training. Since the mask-based patch-level training (EVA) is proven to be unsatisfactory (Table 3), distilling features matches our demands exactly because we can use both CLIP and patch-level distillation target.

Reviewer comment (Experiments):
The empirical study on the effectiveness of the proposed visual tokenizer is insufficient, not to mention some unfairness in the comparisons with previous works. To solidly prove the generality of the proposed distillation technique, the authors should provide more benchmarking results across a wide range of tasks. I doubt distilling on ImageNet-1k is enough to achieve the goal.

Author response:
**[More benchmarking results and generality of visual tokenizer]** 
We provide the evaluation comparison on the SEED-Bench [4], which is a recently released MLLM benchmark and built upon web-images in CC3M . The results show that FD can still improve the performance on the web image benchmark across a wider range of tasks, especially on fine-grained understanding tasks such as instance identity, location, and counting.

|			 |Scene Understanding|Instance Identity|Instance Location|Instance Attributes|Instance Counting|Spatial Relations|Instance Interactions|Visual Reasoning|Avg|
|---|---|---|---|---|---|---|---|---|---|
|GVT wo FD|41.26|34.30|31.40|**29.84**|34.81|**32.98**|**31.96**|50.75|35.93|
|GVT	      |**41.74**|**35.50**|**31.79**|29.45|**36.17**|31.96|**31.96**|**51.06**|**36.20**|

These results show that our visual tokenizer is effective at various vision-language tasks. On the other hand, the effectiveness of FD on various vision tasks has been shown in [3].

Reviewer comment (Experiments):
Why the authors did not use CLIP released by OpenAI as the teacher, but rather used EVA-CLIP as the teacher model, despite knowing that it is inferior to CLIP and other models?

Author response:
We would like to clarify that EVA [1] and EVA-CLIP [2] are two different models. EVA is a Mask-Image-Modeling pretrained model which uses CLIP feature as region supervision, while EVA-CLIP is a contrastive vision-language pretrained model. In essence, EVA is more of an MAE model while EVA-CLIP is more of a CLIP model. Table 3 shows that **EVA** is less useful but we use **EVA-CLIP** as a stronger version of CLIP for feature distillation to achieve a strong MLLM. The comparison between CLIP and EVA-CLIP is shown as follows:

||VQA|Caption|COCO-OC|COCO-MCI|VCR-OC|VCR-MCI|Avg|
|---|---|---|---|---|---|---|---|
|CLIP|47.7|64.2|45.5|**88.0**|34.5|68.8|58.1|
|EVA-CLIP|**48.1**|**65.3**|**46.0**|**88.0**|**34.6**|**69.1**|**58.5**|

The results show that EVA-CLIP is generally better than CLIP in various tasks.