Paper: Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR-2024)

Reviewer comment (Presentation):
Nit: the table and figure fonts are very small and hard to see.

Author response:
Indeed, we fix this in the revised paper

Reviewer comment (Evaluation):
See above question about automated evaluations for instruction following.

Author response:
We understand this concern, and agree that quantitative evaluation is indeed important to support our claims regarding Instruction following. To complete our evaluation, and as suggested by the reviewer, we conduct a quantitative evaluation with GPT-4. 

* Setup: for each question-instruction pair we provide 2 answers to GPT-4: one generated by our evaluated models, and another by GPT-4 itself, and ask it to provide scores for these answers. The reported scores signify the score of our model w.r.t GPT-4 (100% means the evaluated model has the same quality as GPT-4).

* Results: the results in Fig.5 (in the revised paper) support our previous findings; models that are not instruction tuned (IDEFICS and OFv2-9B) struggle to follow user instructions, and ICL improves in this regard, making them more helpful.

Reviewer comment (Experiments):
While the modifications to ICL yield improvements in open-sourced LMMs, it would be advantageous to evaluate these findings in the context of models with >9B parameters, as these larger models are prevalent in the state-of-the-art (SOTA) and also have been shown to exhibit different trends than the smaller models.

Author response:
* Very large models are indeed known to possess some emergent abilities not present in smaller models. However, models up to 9B parameters also possess such abilities and attain SoTA results on most of the benchmarks. We didn’t include larger models in the paper because of the computation cost associated with evaluating such models. 
* To answer this point, we complete our evaluation by considering the largest open source LMMs (IDEFICS-80B and IDEFICS-80B-Instruct); models up to 80B parameters (that can fit on 8GPUs A100). We update the figures in the revised paper. Our findings remain the same, with slight modifications: These models (80B) perform generally better than the 9B-parameter ones, however our claims regarding each axis remain mostly the same.

* The claims that are slightly modified are as follows:
    - Object hallucinations: 80B-parameter models also suffer from hallucinations, similar to smaller models. However, they suffer less from hallucination amplification when tested with ICL.
    - Compositionality: seemingly 80B-parameter models perform significantly better on the CREPE benchmark. However, due to the importance of this axis, we deepen our study and consider a more challenging benchmark (SugarCREPE), with less biases to exploit (results in Appendix F.2). On this benchmark, we show that indeed, all models including 80B-parameter ones do not acquire compositional ability. Thus, our claim holds here as well.

Reviewer comment (Evaluation):
A more comprehensive discussion on why longer CL (32 & 64) does not yield improvements across various axes would enhance the understanding of the models' behaviors.

Author response:
This is an interesting observation that is worth elaborating and be included in the revised paper (Appendix A in the revised paper). The performance saturation after many shots is not only seen in our study, but also in the original work of OpenFlamingo and IDEFICS. For example, in Fig.6 in the OpenFlamingo paper; the VQA accuracy  saturates or even degrades  after 4/8 shots. Similarly for IDEFICS, but slightly better  (https://huggingface.co/HuggingFaceM4/idefics-9b-instruct). 

There is multiple reasons for why multimodal ICL is not as effective as in LLMs, we list the following potential ones:
   - Training data:  the multimodal datasets are still an order of magnitude smaller than those for LLMs. In addition,   the web documents used to train such models do not contain many interleaved image-text pairs (a lot less than 32), which might hinder the ability of the model to generalize to larger number of in-context demonstrations
   - Model scale: The trainable parameters during pretraining, are relatively small (<15B), and acquiring better ICL ability might require training more parameters for more iterations. 

Finally, we would like to highlight the lack of in depth analysis of ICL in the context of LMMs, which we keep for future work.

Reviewer comment (Presentation):
The text in all illustration figures is very difficult to read, and Table 3 is also challenging to interpret due to the small font size.

Author response:
We enhance the readability of the figures in the revised paper, in particular Table .3, Fig.1 and Fig.5.

Reviewer comment (Evaluation):
With three random trials conducted, it remains unclear whether the reported results in the tables represent mean or median values.

Author response:
The reported results are the average over 3 runs as mentioned in the Implementation details section (“We repeat each experiment 3 times and report the averaged results…”).. We first omitted adding the variance to improve the readability of the paper. Now, we added a table (Tab.5) in the appendix showing that the variance is indeed small.

Reviewer comment (Evaluation):
The results lack confidence intervals -- in general there is very large variance between the numbers. For instance in Table 2 and 3, there is large variance of a single method between a different number of shots which seems larger than the reported gains of the proposed method.

Author response:
First, we didn’t report the small confidence intervals to improve the readability of the graphs. As requested, we now add the standard deviation in the revised paper (Tab.5 in Appendix) for 2 representative models. Due to  evaluating on a sufficiently large number of examples (e.g. 8K for abstention) The variance is small, for example:

* For OH with IDEFICS-9B 16-shots the CIDEr/CHs/CHi are 102.19/9.37/6.88 (avg) and 0.24/0.21/0.12 (std), 
* Similarly for OFv2-9B Abstention the Acc/Absurd Acc/Absurd F1 are  46.83/77.84/51.80 (avg) and 0.35/0.03/0.31 (std).

This reveals that our scores are stable. In addition, the variance is very small compared to the change in scores for different numbers of shots.  This further supports our claims and findings in the paper. 

Regarding the variance across the number of shots; comparing different methods, and comparing the same method across different numbers of shots are two different things and should not affect the interpretation of the significance of our approach, as our goal is to compare X-ICL to ICL for a fixed number of demonstrations.

Reviewer comment (Evaluation):
It would be good to report quantitative results for each result in the paper.

Author response:
* We  agree with the reviewer on the importance of providing quantitative results for all benchmarks. In the paper we initially provide scores for all models/benchmarks, except for Instruction following, due to the complexity of evaluating this task quantitatively. To complete our evaluation we now provide scores also for this task automatically by using the now available GPT-4.

* The setup is as follows; for each question-instruction pair we provide 2 answers to GPT-4: one generated by our evaluated models, and another by GPT-4 itself, and ask it to provide scores for these responses. The reported scores signify the score of our model w.r.t GPT-4 (the higher the better). 

* The results in Fig.5 (in the revised paper) support our previous findings; models that are not instruction tuned (IDEFICS and OFv2-9B) struggle to follow instructions, and ICL improves in this regard, making them more helpful.

Reviewer comment (Novelty):
I had difficulty understanding the main differences between the new in-context learning methods presented in the paper and the performances seem minor.

Author response:
* To prioritize more important messages (Section 2), we provide more thorough discussions and details about the evaluation of LMMs and the study of ICL. As for the last part (X-ICL) and due to space limitation, we provide more explanations about different ICL variants in Appendix G. 

* Regarding the minor improvements on some benchmarks, we would like to emphasize that X-ICL is first without training (almost not cost) and second, very simple to implement (no complex procedure or tedious prompt engineering). With such simple methods we are able to show significant improvements on some benchmarks. Improving the performance might require more complex ICL variants which we keep for future work.

Reviewer comment (Experiments):
The existing evaluated models seem out of date -- it would be good to compare with models such as BLIP or LLava, MiniGPT-4 or GPT4V.

Author response:
* We kindly disagree with the reviewer considering the evaluated models outdated, OpenFlamingo and IDEFICS are very recent and not even published yet (IDEFICS will be published in Neurips 2023 and OpenFlamingo still a preprint).

* Besides, our choice is not just based on model size, but rather on the ICL ability of the models. As far as we know (before the submission deadline) OF and IDEFICS are the best open-source models with good ICL ability, which is not the case for BLIP and LlaVA.   Specifically, BLIP and LlaVA are mostly trained on image-text pairs and pass a significant number of tokens to the input of the LLMs, which limit the ability to do ICL, as well as to do ICL with a large number of shots (due to the LLM context limit).

* To try to address the reviewer's concerns, we instead  complete our pool of models with the recent open-source IDEFICS-80B-instruct (as well as IDEFICS-80B). This model was introduced after the OBELICS [1] and behaves well with ICL. The evaluation of these models support further our findings (please also check the answer to rev dAEJ).

* The proposed work and the code that will be released  are meant to facilitate the evaluation of future LMMs on current and new benchmarks.

[2] Laurençon, Hugo, et al. "OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents." NeurIPS 2023.

Reviewer comment (Reproducibility):
Can the authors provide psuedocode for each of the in-context learning methods that are proposed?

Author response:
* The authors will release the full code to reproduce the results. As for now, we provide a link to the full code here: https://anonymous.4open.science/r/EvALign-ICL-8DC9/README.md. The different ICL variants can be found in open_flamingo/eval/ in the following files; caption_utils.py/vqa_utils.py and itm_utils.py

* In addition, we provide the necessary details to implement these variants in the paper in form of equations (eq. 2 and 3), illustration figures (Fig. 13/15/16), in addition to more details in Appendix G. For ICL, we follow the typical pipeline used in IDEFICS and openflamingo, that you can also check in https://github.com/mlfoundations/open_flamingo

Reviewer comment (Theory):
Why does the modification to in-context learning of presenting both positive and negative demonstrations correspond to CoH in context learning?

Author response:
This is inspired by CoH [2], in which, the authors pass **during training** both positive and negative examples to the model as input (instead of passing only the positive one).  In our case, we do the same, but instead of training the model, we do it during ICL. This shows that those approaches, first introduced for training, can actually work at test time with ICL, thus improving efficiency and practicality. 

[2] Liu, Hao, Carmelo Sferrazza, and Pieter Abbeel. "Languages are rewards: Hindsight finetuning using human feedback." arXiv 2023.

Reviewer comment (Novelty):
While the paper addresses an essential aspect of LMMs' functionality, it falls short in pioneering novelty. It does not introduce new datasets or unique evaluation methodologies, which limits its contribution to the field.

Author response:
Our evaluation is novel in multiple ways:
* The first work that highlights the limitations of LMMs from 3B to 9B (and now even 80B in the revised paper, see R.dAEJ)  on different and very important aspects, beyond mere accuracy on academic benchmarks.
* The first work studying multimodal-ICL and its effect on addressing the limitations of LMMs, which provide a critical analysis of a de facto approach used to adapt LMMs and LLMs.
* The first work proposing new multimodal-ICL variants that bring improvements on several benchmarks, despite their simplicity.
We believe that such work and effort is important for the community to start focusing on fixing and addressing such limitations, instead of just saturating academic benchmarks.

Reviewer comment (Evaluation):
The potential impact of the proposed ICL variants is not sufficiently analyzed.

Author response:
We prioritize other and more important messages in the paper, and due to lack of space we put more details about X-iCL in Appendix G. Please also check the general response to see our additional analysis and more details about how we frame the paper and which messages are important to convey and study. 

The main objective of adding this section at the end of the paper  (Sec.3 X-ICL) is to open up a potential direction to work on aligning and alleviating the limitations of LMMs more efficiently (as compared to training) via prompting and ICL. We keep more thorough study about X-ICL as well as more complex variants for future work.

Reviewer comment (Reproducibility):
How is the selection process for demonstrations in context $C$ determined?

Author response:
The demonstrations are selected randomly as explained for instance in Section 2 (Implementation detail), Sec. 2.5, Appendix D
Our selection also follows the standard setup in IDEFICS and Open Flamingo.

Reviewer comment (Evaluation):
Could you specify the evaluation metrics used for each of the five axes and explain why the metrics can measure the axes?

Author response:
We use standard and well-established metrics for each axis, as well as citations for each paper that explain why the underlying metric measures a given axis. The metrics for each benchmark/dataset are detailed in the “Benchmark” paragraph of each subsection of Section 2. For example, in Sec.2.1, we state: “we report the CHAIRs metric (Rohrbach et al., 2018) comparing the objects referred in the generated captioning to those actually in the image”. Due to space constraint, those metrics are further described in details in Appendix. E.