Paper: ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks (ICLR-2024)

Reviewer comment (Novelty):
Limitations in novelty to some extent: the formulation of multiple-choice is widely used in previous work like MMBench [1]; the proposed generation and likelihood evaluation method are also used in previous benchmarks, like VisualGPTScore [2] proposed using likelihood of generating references conditioned on images and prompts to do multiple-choices tasks.

Author response:
**The core contribution of ReForm-Eval is a strategy for re-formulating existing task-oriented datasets to suit the evaluation of LVLMs**. The reason for choosing multiple-choice questions is that this format is in line with the evaluation needs of LVLMs to output in free-form texts. At the same time, based on the findings that current LVLMs have weak capability in following multiple-choice instructions, we further assist the evaluation through both black-box (text-only in-context samples) and white-box (likelihood) methods. Thus, **ReForm-Eval exhibits a versatile and efficient way to evaluate LVLMs by combining these individual components in a reasonable way.**

Reviewer comment (Evaluation):
Should we use generation metric, likelihood metric, or both of them when using the benchmark?

Author response:
If possible, **we recommend using both approaches for comparison.** The **black-box strategy is more versatile** but requires higher adherence to instructions by models; some models (like BLIVA) require the **white-box method to better reflect their multimodal capabilities**, and it offers higher stability and computational efficiency. At the same time, comparing the performance differences between the two strategies can also reveal the instruction-following abilities between models.

Reviewer comment (Novelty):
Some findings presented in this paper are not original findings. For example, the effect of in-context examples and the instability of existing LVLMs have already been discussed in MMBench.

Author response:
1. In-context samples have been widely used during the model training stage to enhance the instruction following ability of models, i.e. MIMIC-IT dataset used in Otter. **To the best of our knowledge,** **ReForm-Eval is the first attempt to utilize in-context samples to** **assist the** **evaluation of LVLMs****.** In this paper, **our novel findings is that** complete image-text pairs are not necessary as in-context samples; **text-only in-context samples can provide effective guidance**, helping LVLMs to answer multiple-choice questions in the correct format.
2. The analysis of instability in MMBench is limited to the impact of option order perturbations. **In ReForm-Eval, we comprehensively consider various perturbations****:** not only shuffling option order, but also applying different instructions and option markers. Furthermore, we **quantitatively measure the instability of models in the form of entropy**. Above are the original contributions of ReForm-Eval in terms of instability.

Reviewer comment (Experiments):
Fine-grained recognition, which is a substantial component of Fine-grained perception, is not included in the fine-grained perception tasks.

Author response:
We believe this is a misunderstanding due to the different interpretations of term "fine-grained". **In ReForm-Eval, fine-grained perception refers to the perception of semantic information at the level of local objects.** On the other hand, fine-grained image classification tasks (**also known as fine-grained recognition**) focus on distinguishing fine-grained categories, which involve perception at the image level. Therefore, in ReForm-Eval, **such tasks are categorized as image classification tasks in coarse-grained perception**, such as Flowers102 and Pets37.

Reviewer comment (Presentation):
For Figure 10, why are the proportions of 'A' and 'B' in Ground-truth Option Distribution not the same? Does that mean there exist questions with only one option?

Author response:
Response:  In this dataset, there are no questions with only one option. The distribution of correct answers has a certain level of randomness due to the shuffle, so the proportions of A and B are not exactly the same.

Reviewer comment (Writing):
Typo in Page 8 Line 1, MSOCO.

Author response:
Response: Thank you for pointing out the error. We have corrected this typo.

Reviewer comment (Experiments):
Reform-Eval is a large dataset containing over 500,000 evaluation instances. When doing the sub-sampling, do the authors do it uniformly or sample evaluation instances from each benchmark with different probability to improve the data balance? Besides, have the authors studied if the shrink of the dataset size will change the evaluation results? Can we use an even smaller subset for evaluation?

Author response:
- **Detailed explanation of our sampling method:** Since the Reform-Eval covers 61 benchmarks and contains more than 500,000 evaluation samples, we employ ***a balanced sampling strategy*** to enhance the robustness of the evaluation. Within each benchmark, we sample 10% data based on the distribution of the original data **except for three cases**: (1) when the size of the original benchmark is less than 1000, we keep the whole benchmark for a stable evaluation; (2) when the original benchmark has more than 10,000 evaluation samples, we filter the data and then conduct the sampling process; (3) for benchmarks used in Multi-turn Dialogue dimension, we retain all evaluation samples as only two datasets are included and the total sample volume is moderate in this dimension (~3000).

- **Why we sample at a rate** **of** **10%**: We analyze whether the shrink or expansion of the dataset size will change the evaluation results, by conducting several experiments on the VQAv2 benchmark and Flowers102 benchmark (the evaluation sample size is 21441 and 818, respectively). Rebuttal Table 5 demonstrates that the more data sampled, the more consistent they are with the evaluation on the complete dataset. **A 10% sampling ratio can achieve a good balance between the evaluation efficiency and consistency.** 

- **Rationality of our sampling method**: As shown in Rebuttal Table 6, for larger datasets, the sampling ratio has less impact on the results; for smaller datasets, different sampling ratios cause the results to fluctuate  dramatically. **To ensure the stability of the evaluation, we generally perform balanced sampling for larger datasets and retain the entire dataset for smaller datasets.** 

Rebuttal Table 5: Evaluation results under different sampling ratios on the VQAv2 and Flowers102 benchmark. We derive the results of different models on test sub-benchmarks under each sampling ratio, and calculate the correlation coefficient $\rho$ and average absolute deviation $\bar{d}$ of these results compared with the results on the complete test benchmark.

| **Dataset**    | **Metric**  | **Generation** |        |         |         |          | **Likelihood** |         |         |         |          |
| -------------- | ----------- | -------------- | ------ | ------- | ------- | -------- | -------------- | ------- | ------- | ------- | -------- |
|                |             | **1%**         | **2%** | **10%** | **20%** | **100%** | **1%**         | **2%**  | **10%** | **20%** | **100%** |
| **VQAv2**      | $\rho$    | 0.9861         | 0.9948 | 0.9989  | 0.9996  | 1        | 0.9689         | 0.9857  | 0.9970  | 0.9991  | 1        |
|                | $\bar{d}$ | 3.1483         | 1.7075 | 0.5550  | 0.3658  | 0        | 2.6450         | 2.1167  | 0.8725  | 0.4958  | 0        |
| **Flowers102** | $\rho$    | 0.9575         | 0.9559 | 0.9794  | 0.9336  | 1        | 0.7984         | 0.7861  | 0.9131  | 0.9727  | 1        |
|                | $\bar{d}$ | 8.5738         | 9.0256 | 4.3775  | 1.6994  | 0        | 12.1756        | 10.6850 | 3.2488  | 2.9275  | 0        |

Rebuttal Table 6: Variance of accuracy(%) under different sampling ratios. We repeatedly sample a certain percentage of the data for 10 times. We derive the accuracy each time and then compute the variance for each model. The final variance value is averaged across the models.

| **Dataset**    | **Size** | **Generation** |        |         |         |          | **Likelihood** |        |         |         |          |
| -------------- | -------- | -------------- | ------ | ------- | ------- | -------- | -------------- | ------ | ------- | ------- | -------- |
|                |          | **1%**         | **2%** | **10%** | **20%** | **100%** | **1%**         | **2%** | **10%** | **20%** | **100%** |
| **VQAv2**      | 21441    | 5.22           | 2.90   | 0.44    | 0.19    | 0        | 8.63           | 5.06   | 0.81    | 0.26    | 0        |
| **Flowers102** | 818      | 169.79         | 85.60  | 18.83   | 6.62    | 0        | 243.01         | 174.59 | 17.79   | 10.66   | 0        |

Reviewer comment (Evaluation):
Using CIDEr to evaluate visual descriptions is not optimal, especially for models that intend not to generate concise descriptions, such as LLaVA, and it benefits models that are tuned by dataset, such as coco-caption, whose ground truth descriptions are in a shorter format.

Author response:
We chose CIDEr as an automated evaluation metric following BLIP-2. In the updated version of our paper (see Table 20), we have supplemented other generation metrics including BLEU-4, Meteor and Rouge-L for a comprehensive evaluation. Here is a quick look:

Rebuttal Table 4: Evaluation results on visual description based on CIDEr, BLEU-4, Meteor and Rouge-L. The evaluation dataset is NoCaps for zero-shot evaluation.
| **Model**          | **CIDEr** | **BLEU-4** | **Meteor** | **Rouge-L** |
| ------------------ | --------- | ---------- | ---------- | ----------- |
| **BLIP-2**         | 83.57     | 37.18      | 27.09      | 49.34       |
| **Lynx**           | 77.31     | 30.52      | 26.26      | 47.30       |
| **MiniGPT4**       | 58.71     | 22.84      | 24.99      | 40.83       |
| **mPLUG-Owl**      | 48.43     | 17.35      | 21.68      | 37.66       |
| **LLaVA_V**        | 42.43     | 17.39      | 21.67      | 36.27       |
| **InstructBLIP_V** | 30.18     | 7.71       | 13.48      | 25.13       |

Reviewer comment (Experiments):
The methods to generate appropriate negative options in visually grounded reasoning and multi-turn dialogue may not be reliable. The information of question and answer may not be sufficient to generate reasonable negative options with ChatGPT.

Author response:
**Q3: Lack of rational analysis of using ChatGPT to generate distractors based on textual QA pairs.**

Response:  Thank you for the feedback. Here, we assess the rationality of distractors on two levels:

* **Reliability**: whether distractors face false-negative issues; 
* **Difficulty**: whether negative options are challenging and not overly naive. 

According to the Rebuttal Table 3 in the common response, **ChatGPT-generated distractors have a low false-negative rate in terms of reliability**. Regarding difficulty, we find that the performance of all models is not satisfactory (the best model achieves an accuracy of 55%), indicating that **ChatGPT-generated distractors are challenging**. Introducing additional image-related information, such as captions and objects, would require extra information or the use of detection models, incurring higher costs. Therefore, we believe the current approach is relatively reliable and efficient.