Paper: Visual Data-Type Understanding does not emerge from scaling Vision-Language Models (ICLR-2024)

Reviewer comment (Experiments):
A core component of experiments is language-based classification (into one of 27 data-types). However, the authors simply use a short description for each data-type (as in appendix) on the language side. This could simply mean that the model is not being prompted correctly (and the sensitivity of VLMs to prompting is well-known). If this is the case, the assertions of weaknesses (i.e., inability to distinguish data-types) are not generic, but only for the selected set of prompts used in the paper.

Author response:
- **Regarding your first comment on prompting the LMMs and your third question on the reasoning for weakness.** We agree that reasonably prompting the LMMs is crucial for prompt-based model evaluation. Following your suggestions, we now have run additional experiments, verifying that the results reported in our paper did not emerge from a peculiarity in the prompts we used. 
1. We created two additional sets of alternate prompts for the 27 data-types capturing language diversity. Inspired by VisDesc ([Menon and Vondrick ICLR, 2023](https://arxiv.org/abs/2210.07183)) and CuPL ([Pratt et al ICCV, 2023](https://arxiv.org/abs/2209.03320)), we prompted GPT-4 to suggest alternative prompts for each of the original data-type prompts and manually verified that they actually capture the correct data-type concept. For instance, for the PATCH_AND_RESHUFFLE data-type, in our initial experiments we probed VLMs with the prompt “This is a patched and reshuffled image of an animal”. For the two alternative prompt-sets we probe them with “This is a scrambled image of an animal.” and “This is a jumbled image of an animal.” (the full set of alternate prompts for all data-types is listed in Appendix B.3). Evaluating a representative subset of models on SyntheticTypeIdent revealed highest model performance for the prompts we originally chose, while the alternative prompts performed slightly worse (see new Appendix C and Figure 7 for results). Further, we were curious how the newer LLaVA 1.5 model you explored would perform on our task and included it into our systematic evaluation (both the 7B and 13B variants). In line with what you found in your experiments with the web-API, we found an improvement over LLaVA 1.0 (see Figure 7). At the same time, the absolute performance is comparable to the other LMMs tested.
2. To further extend this analysis, we followed your suggestion and used the GPT-3 prompting method introduced by the paper you cited (VisDesc ([Menon and Vondrick ICLR, 2023](https://arxiv.org/abs/2210.07183))), to generate a large set of data-type descriptors (5-7 descriptors per data-type). To probe C-VLMs with these descriptors, we followed the evaluation scheme by VisDesc: we averaged the corresponding text embeddings for each data-type and predicted the data-type whose embedding had the highest similarity with the test image. LMMs cannot be probed by comparing to averaged embeddings. Hence, we averaged the log-likelihood across descriptors for each data-type, akin to an evaluation strategy used for OpenFlamingo (see [Github](https://github.com/mlfoundations/open_flamingo/blob/655f693fbfa04cd6e9a987d960654624d48917cf/open_flamingo/eval/evaluate.py#L83C1-L87C2)). This much more sophisticated prompting method led to similar results as in our main analysis and the alternative prompt experiments (see new Appendix section C and Figure 8).

In conclusion, while the exact prompting style showed some effect on model performance, overall prompt-based results on our data-type identification tasks (C-VLM outperforming LMMs, LMMs performing generally poor, only marginal scaling behaviour) robustly stayed the same across prompting strategies.

Reviewer comment (Evaluation):
For simple verification, I took 10 random images, applied the patch&reshuffle operation, and passed both original and augmented to the online LLava API (https://llava.hliu.cc — this is one of the models evaluated in the paper / API uses newer checkpoint). While it did not generate the expected term "patch&reshuffle", it generated outputs for the augmented images different to the original, consistently involving terms like "collage, mosaic, collage art style" (words somewhat synonymous to augmentation) which indicate some awareness of the model to this patch&reshuffle operation. The original images were not described with these keywords. However, according to Figure 4 in the paper, the best LMM (LLava is one of those evaluated) has 0 informedness about patch&reshuffle. For this case, either the metric of informedness or the evaluation protocol does not well represent the actual abilities of evaluated models like LLava.

Author response:
- **Regarding your first comment on prompting the LMMs and your third question on the reasoning for weakness.** We agree that reasonably prompting the LMMs is crucial for prompt-based model evaluation. Following your suggestions, we now have run additional experiments, verifying that the results reported in our paper did not emerge from a peculiarity in the prompts we used. 
1. We created two additional sets of alternate prompts for the 27 data-types capturing language diversity. Inspired by VisDesc ([Menon and Vondrick ICLR, 2023](https://arxiv.org/abs/2210.07183)) and CuPL ([Pratt et al ICCV, 2023](https://arxiv.org/abs/2209.03320)), we prompted GPT-4 to suggest alternative prompts for each of the original data-type prompts and manually verified that they actually capture the correct data-type concept. For instance, for the PATCH_AND_RESHUFFLE data-type, in our initial experiments we probed VLMs with the prompt “This is a patched and reshuffled image of an animal”. For the two alternative prompt-sets we probe them with “This is a scrambled image of an animal.” and “This is a jumbled image of an animal.” (the full set of alternate prompts for all data-types is listed in Appendix B.3). Evaluating a representative subset of models on SyntheticTypeIdent revealed highest model performance for the prompts we originally chose, while the alternative prompts performed slightly worse (see new Appendix C and Figure 7 for results). Further, we were curious how the newer LLaVA 1.5 model you explored would perform on our task and included it into our systematic evaluation (both the 7B and 13B variants). In line with what you found in your experiments with the web-API, we found an improvement over LLaVA 1.0 (see Figure 7). At the same time, the absolute performance is comparable to the other LMMs tested.
2. To further extend this analysis, we followed your suggestion and used the GPT-3 prompting method introduced by the paper you cited (VisDesc ([Menon and Vondrick ICLR, 2023](https://arxiv.org/abs/2210.07183))), to generate a large set of data-type descriptors (5-7 descriptors per data-type). To probe C-VLMs with these descriptors, we followed the evaluation scheme by VisDesc: we averaged the corresponding text embeddings for each data-type and predicted the data-type whose embedding had the highest similarity with the test image. LMMs cannot be probed by comparing to averaged embeddings. Hence, we averaged the log-likelihood across descriptors for each data-type, akin to an evaluation strategy used for OpenFlamingo (see [Github](https://github.com/mlfoundations/open_flamingo/blob/655f693fbfa04cd6e9a987d960654624d48917cf/open_flamingo/eval/evaluate.py#L83C1-L87C2)). This much more sophisticated prompting method led to similar results as in our main analysis and the alternative prompt experiments (see new Appendix section C and Figure 8).

In conclusion, while the exact prompting style showed some effect on model performance, overall prompt-based results on our data-type identification tasks (C-VLM outperforming LMMs, LMMs performing generally poor, only marginal scaling behaviour) robustly stayed the same across prompting strategies.

Reviewer comment (Evaluation):
"For a fair comparison, we evaluated LMMs by log-likelihood scoring" — please explain this in detail in the main text (or at least appendix) without directing the reader to other papers (in fact the directed papers also re-direct to different papers). This evaluation is crucial to understand all the reported analysis. Please explain it clearly.

Author response:
- **Regarding LMM evaluation details.** We thank you for pointing out that this information was not clearly included in our paper. To compute the zero-shot performance of C-VLMs, we computed the matching score of each image $I$ with each of the 27 data-type text prompts $\{D_1, D_2, …, D_{27}\}$ and predicted the data type with the highest score, i.e. $\text{predicted data-type} = \arg \max_{i\in{1, …, 27}} I_{\text{enc}}(I) \cdot T_{\text{enc}}(D_i)$. Since LMMs are auto-regressively trained, evaluating them in the same way is impossible. For a fair comparison of LMMs to C-VLMs, we instead computed the LMMs log-likelihood of a prompt joining the image and the data-type text prompts. By default, we prompted LMMs to compute the log-likelihood for the query prompt $P_i =$ “<image> Q: Describe the image. A: $D_i$”, where $D_i$ is replaced by the specific data-type text prompt. Specifically, we tokenized the prompt $P_i$ and retrieved the model log-probabilities for each input token. Then, we summed the log-probabilities across tokens yielding the final log-probability for the particular data-type, $L_i(P_i(I, D_i))$. We predicted the data-type of an image $I$ by the highest log-probability across data-types, $\text{predicted data-type} = \arg \max_{i\in{1, …, 27}} L_i(P_i(I, D_i))$. Some LMMs were trained with a particular prompt pattern for instruction tuning, and for those, we used their specific prompt template $P$ (see Appendix B.5 for full list).

Reviewer comment (Experiments):
The size of the model (i.e. 7B) used for in-context learning experiments might be too small to test the capability of in-context learning. In-context learning with larger models might work so I think it would be better if the authors could clarify this point.

Author response:
- **Regarding in-context learning with larger models.** For Otter, which we previously investigated (Figure 6b), unfortunately no in-context learning model versions larger than the investigated 7B model exist. Thus, we repeated the same experiments as for Otter with the LLaVA models, where both 7B and 13B variants are available. Interestingly, both LLaVA versions underperformed Otter-LLaMA7B and Otter-MPT7B, and surprisingly, LLaVA-13B even underperformed its smaller LLaVA-7B version (new Appendix Section H.3 and Figure 11). This experiment further corroborates the claim of our paper that in-context learning seemingly does not provide benefits for improving the performance of LMMs like Otter and LLaVA on data-type identification. Thank you again for this great suggestion.

Reviewer comment (Evaluation):
For Section 4.1, when assessing the accuracy of LMMs, aren't there more than one correct answer for data type description? For example, "pencil sketch" could be "pencil drawing", "black and white drawing" etc and "Origami style" could be "low-poly style" etc. How did you deal with these?

Author response:
- **Regarding the comment about prompt sensitivity.** Thank you for this great question! In our initial zero-shot results, we had used a default, fixed set of prompts for all our experiments. However, we now have run additional experiments to check for prompt sensitivity, verifying that the results reported in our paper did not emerge from a peculiarity in the prompts we used.
1. We created two additional sets of alternate prompts for the 27 data-types capturing language diversity. Inspired by VisDesc ([Menon and Vondrick, ICLR, 2023](https://arxiv.org/abs/2210.07183)) and CuPL ([Pratt et al, ICCV, 2023](https://arxiv.org/abs/2209.03320)), we prompted GPT-4 to suggest alternative prompts for each of the original data-type prompts and manually verified that they actually capture the correct data-type concept. For instance, for the PATCH_AND_RESHUFFLE data-type, in our initial experiments we probed VLMs with the prompt “This is a patched and reshuffled image of an animal”. For the two alternative prompt-sets we probe them with “This is a scrambled image of an animal.” and “This is a jumbled image of an animal.” (the full set of alternate prompts for all data-types is listed in Appendix B.3). Evaluating a representative subset of models on SyntheticTypeIdent revealed highest model performance for the prompts we originally chose, while the alternative prompts performed slightly worse (see new Appendix C and Figure 7 for results).
2. To further extend this analysis, we followed the GPT-3 prompting method introduced by VisDesc ([Menon and Vondrick, ICLR, 2023](https://arxiv.org/abs/2210.07183)), to generate a large set of data-type descriptors (5-7 descriptors per data-type). To probe C-VLMs with these descriptors, we followed the evaluation scheme by VisDesc: we averaged the corresponding text embeddings for each data-type and predicted the data-type whose embedding had the highest similarity with the test image. LMMs cannot be probed by comparing to averaged embeddings. Hence, we averaged the log-likelihood across descriptors for each data-type, akin to an evaluation strategy used for OpenFlamingo (see [Github](https://github.com/mlfoundations/open_flamingo/blob/655f693fbfa04cd6e9a987d960654624d48917cf/open_flamingo/eval/evaluate.py#L83C1-L87C2)). This much more sophisticated prompting method led to similar results as in our main analysis and the alternative prompt experiments (see new Appendix section C and Figure 8).

In conclusion, while the exact prompting style showed some effect on model performance, overall prompt-based results on our data-type identification tasks (C-VLM outperforming LMMs, LMMs performing generally poor, only marginal scaling behaviour) robustly stayed the same across prompting strategies.