Paper: ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models (ICLR-2024)

Reviewer comment (Novelty):
I am not certain if it is fair to incorporate current MLLM benchmarks into ChEF. These benchmarks have taken a significant amount of time to develop, so what is the core contribution of ChEF?

Author response:
### Q1: What is the core contribution of ChEF?

We sum up the core contribution of our work in section 4 of our overall response. Various evaluation pipelines and criteria make fair comparison within the same evaluation pipeline a considerable challenge for MLLMs. ChEF effectively addresses these issues, enabling unified evaluation, and providing many interesting observations provided in our paper and supplementary materials. ChEF also propose evaluation on several capabilities that beyond visual performance, which have not been evaluated before. 
Additionally, ChEF is scalable, enabling users to extend their recipes for each component and supporting various evaluation pipelines of different dimensions of capabilities. These contributions assist users in assessing MLLM performance and guiding the improvement of capabilities in MLLMs.

Reviewer comment (Experiments):
Besides in-context learning, ChEF only evaluates single-image input. However, the comprehension of multi-image input is also an important assessment dimension for MLLMs.

Author response:
### Q2: The comprehension of multi-image input is also an important assessment dimension for MLLMs.

We are very grateful for your suggestion of this important capability dimension. However, it's important to understand that currently, very few open-source models support multiple image inputs, especially for multi-image VQA tasks, which require the insertion of image tokens into text tokens. This feature is not supported by most open-source models. Nevertheless, in ChEF, we rapidly implemented a multi-image evaluation recipe and conducted evaluations on the Winoground[1] dataset for four models that support multi-image input. Winoground is a dataset that proposes a task of correctly matching two given images and captions, which can be used to evaluate MLLMs' understanding of text-to-image references. The results are shown below:

|**Setting**| **Model** | **Text** | **Image** | **Group** |
| :-------: | :-------: | :-------: | :-------: | :-------: |
|MLLM multi-image |**mPLUG-owl**|	36.25|	38|	31.25|
|MLLM multi-image |**MiniGPT-4**|	26|	36	|22.75|
|MLLM multi-image |**Kosmos-2**|	25.5|	32.5|	20.75|
|MLLM multi-image |**Otter**|	22.75|	33.25|	18.75|
|MLLM single-image|**PALI**| 	46.5|	38|	28.75|
|MLLM single-image|**Blip-2** |	44|	26|	23.5|
|CLIP-based       |	**VQ2**|	47|	42.2|	30.5|
|	              / |**MTurk Human**| 	89.5|	88.5|	85.5|
|	              / |**Random Chance**|	25|	25|	16.67|


Winoground-Text/Image/Group are text, image and group score metrics proposed in Winoground. We prompt the MLLM with "Does image a match with the text b?" and calculate the probability that MLLM output "yes". Different from previous works where similarity was computed for individual image-text pairs, such as VQ2[2], PALI[3], and Blip-2[4]. In our experiment, we take two images and two captions together with a question as input to the MLLM models. Therefore, comparing the results directly with previous works may not be entirely fair due to the differences in experimental setup.

The results indicates that the current MLLM models are not ideal in handling multiple images. It's noticeable that Otter, which was specially trained on ICL data, exhibits lower performance than the other MLLMs. This might be due to a significant gap between treating each image as an individual case and linking multiple images together in a task. There also exists a certain 'tug of war' in multi-image tasks. 

We are very grateful for your suggestion, which brings the interesting findings. We will incorporate more essential evaluation pipelines in the future. We also hope the community will provide more recommendations. 

[1] Winoground: Probing vision and language models for visio-linguistic compositionality.

[2] What you see is what you read? improving text-image alignment evaluation.

[3] Pali: A jointly-scaled multilingual language-image model.

[4] Bootstrapping language-image pretraining with frozen image encoders and large language models.

Reviewer comment (Theory):
The paper does not justify its choices in a principled manner. The overall framework has an ad-hoc feel to it. While the references are comprehensive, there is not enough logic to back up why those six desiderata for example are chosen and why some others are not. The work comes across as an engineering requirements style work rather than a scientific paper. I am open to being convinced otherwise. The field is moving very fast so just seemingly brute force evaluation of a bunch of models is not going to be helpful.

Author response:
### Q1: There is not enough logic to back up why those six desiderata for example are chosen and why some others are not.

We provide the reason for our choice of these desiderata in section 3 of our overall response. More details of these desiderata are supported in Supplementary Section C. **The principles of the six dimensions we selected are based on a survey and statistical analysis of the original LLM field, as well as the application of MLLM as an AI agent.**

Reviewer comment (Novelty):
What are the insights that drive your work? Please see the comments above on weaknesses.

Author response:
### Q4: What are the insights that drive your work?

In the current development of MLLMs in the open-source community, we have identified that evaluating the capabilities of an MLLM is a challenging task, as claimed in section 2 of our overall response.
Therefore, **we aim to establish an evaluation framework that allows modules with different evaluation criteria to be compatible with each other.** This compatibility facilitates the establishment of various evaluation pipelines for **a unified assessment of models**, leading to many comparative findings. At the same time, we seek to evaluate metrics beyond visual capabilities, such as the trustworthiness and interactivity of MLLMs, to ensure that MLLMs **maintain the inherent capabilities required of a Large Language Model (LLM)**. Hence, we have designed new recipes for evaluating six dimensions of desiderata, leading to many interesting findings. We even discovered that visual capabilities are, to some extent, related to these desiderata, as mentioned in Section 3.5. We hope that our framework and the desiderata will guide the performance improvement of MLLMs, and these intensions drive my work.

Reviewer comment (Reproducibility):
Since the main contribution of this paper is introducing this new evaluation framework, I suggest the authors to add a section describing the system design/implementation of this framework in detail.

Author response:
Thank you for your insightful comments on the ChEF framework. I appreciate your perspective and would like to address your concern about the system design/implementation of ChEF.

The criteria supported in each module, as well as their interrelationships, are shown in Figure 1(a) in our paper. We also provide the functions and details of each module's implementation in the Appendix Section B. To clearly present the entire evaluation pipeline, we provide details of the data flow, explaining how the data from scenario is transformed into input through instruction, and then how the inferencer enables the model to output answers for evaluation.

```
1. Load model and scenario. If ppl configured, each data sample in the scenario will include several predefined negative candidates. These negative candidates, together with the ground truth, form an `answer_pool`.
2. Configure the instruction, inferencer, and metric.
3. Iterate through each sample in the scenario. The instruction transforms each sample into an adaptive input for the model. When evaluating `sample_a`:
    3.1 Get the input question from `sample_a`. For VQA tasks, a specific question is presented, while for other tasks, such as classification, there may be no question.
    3.2 Concatenate the standard query or a specified query from the query pool with the question, along with the input image, to form `input_a`. If ICE is configured, retrieve a specified number of ICE from the scenario. Retrieval methods include random (random selection), fixed (selecting samples from the scenario via configured IDs), top-k images (based on similarity of images to the image of `sample_a`), and top-k text (based on similarity of text to the text of `sample_a`). The retrieval has following steps:
      3.2.1 Retrieve a specified number of ICEs.
      3.2.2 For each sample in ICE, concatenate the query with the question, to align with `input_a`.
      3.2.3 Combine ICE and `input_a`, according to the MLLM's accepted format. For MLLMs that accept multiple images input, combine the entire ICE and `input_a`, while for MLLMs that accept only a single image, combine `input_a` with ICE without images, along with an additional system message.
    3.3 If multi-turn inferencer is used, copy the input and replace the query in each with the corresponding multi-turn query.
4. Feed the inputs into MLLM and obtain the response, by outputing the probability token by token. There are several types of inferencer:
    4.1 Direct inferencer: Choose the word with the highest probability for each token as the output.
    4.2 CoT inferencer: Save the query in input as `original_query`, and replace that with a special query 'Let's think step by step'. Execute step 4.1, and then combine the response with `original_query` and execute 4.1 again.
    4.3 PPL inferencer: Replicate the input, and append each candidate from the `answer_pool` to the query of each input. Feed to MLLM and calculate perplexity, and select the candidate corresponding to the input with the lowest perplexity as the response.
    4.4 Multi-turn inferencer: Iterate the turn and exceute the corresponding inferencer.
5. Save the reponse results and execute the metric.
```
Each component can be implemented to meet specific needs by passing a simple configuration. To more intuitively explain the design and implementation of our framework, we provide the pseudo code aligned with the data flow claimed above. 
```
# step 1
model = get_model(model_name)
scenario = build_scenario(scenario_name, ppl_cfg, **other_cfgs)
# step 2
instruction = build_instruction(instruction_type, prompt_cfg, icl_cfg, **other_cfgs)
inferencer = build_inferencer(inferencer_type, **other_cfgs)
metric = build_metric(scenario_name, **other_cfgs)

answers = inferencer.inference(scenario, model, instruction) # step 3 and 4

def inference(scenario, model, instruction):
    answers = []
    for sample in scenario:
        input = instruction.generate(sample) # step 3
        output = model.inference(input) # step 4
        answers += output
    return answers

def generate(sample):
    question = sample.get('question','') # step 3.1
    # step 3.2
    prompt = get_prompt(prompt_cfg) 
    input = prompt.format(question)
    # step 3.2.1 to 3.2.3
    ice = retriever.retrieve(sample)
    ice = [(image, prompt.format(text)) for (image, text) in ice]
    # input = [prompt_t.format(question) for prompt_t in multi_turn_prompt] # step 3.3
    input = (ice, sample['image'], input)
    return input

def inference(input):
    return model.direct_inference(input) # 4.1
    # return model.CoT_inference(input) # 4.2
    # return model.PPL_inference(input) # 4.3
    # return model.multiturn_inference(input) # 4.4

results = metric.metric(answers) # step 5
```

We sincerely hope these provide sufficient clarity. We will also provide a more detailed tutorial and toolkit in the code release version, and include these details in the final version of our paper after revision.

Reviewer comment (Reproducibility):
It seems that such information is missing in the current draft.

Author response:
Thank you for your insightful comments on the ChEF framework. I appreciate your perspective and would like to address your concern about the system design/implementation of ChEF.

The criteria supported in each module, as well as their interrelationships, are shown in Figure 1(a) in our paper. We also provide the functions and details of each module's implementation in the Appendix Section B. To clearly present the entire evaluation pipeline, we provide details of the data flow, explaining how the data from scenario is transformed into input through instruction, and then how the inferencer enables the model to output answers for evaluation.

```
1. Load model and scenario. If ppl configured, each data sample in the scenario will include several predefined negative candidates. These negative candidates, together with the ground truth, form an `answer_pool`.
2. Configure the instruction, inferencer, and metric.
3. Iterate through each sample in the scenario. The instruction transforms each sample into an adaptive input for the model. When evaluating `sample_a`:
    3.1 Get the input question from `sample_a`. For VQA tasks, a specific question is presented, while for other tasks, such as classification, there may be no question.
    3.2 Concatenate the standard query or a specified query from the query pool with the question, along with the input image, to form `input_a`. If ICE is configured, retrieve a specified number of ICE from the scenario. Retrieval methods include random (random selection), fixed (selecting samples from the scenario via configured IDs), top-k images (based on similarity of images to the image of `sample_a`), and top-k text (based on similarity of text to the text of `sample_a`). The retrieval has following steps:
      3.2.1 Retrieve a specified number of ICEs.
      3.2.2 For each sample in ICE, concatenate the query with the question, to align with `input_a`.
      3.2.3 Combine ICE and `input_a`, according to the MLLM's accepted format. For MLLMs that accept multiple images input, combine the entire ICE and `input_a`, while for MLLMs that accept only a single image, combine `input_a` with ICE without images, along with an additional system message.
    3.3 If multi-turn inferencer is used, copy the input and replace the query in each with the corresponding multi-turn query.
4. Feed the inputs into MLLM and obtain the response, by outputing the probability token by token. There are several types of inferencer:
    4.1 Direct inferencer: Choose the word with the highest probability for each token as the output.
    4.2 CoT inferencer: Save the query in input as `original_query`, and replace that with a special query 'Let's think step by step'. Execute step 4.1, and then combine the response with `original_query` and execute 4.1 again.
    4.3 PPL inferencer: Replicate the input, and append each candidate from the `answer_pool` to the query of each input. Feed to MLLM and calculate perplexity, and select the candidate corresponding to the input with the lowest perplexity as the response.
    4.4 Multi-turn inferencer: Iterate the turn and exceute the corresponding inferencer.
5. Save the reponse results and execute the metric.
```
Each component can be implemented to meet specific needs by passing a simple configuration. To more intuitively explain the design and implementation of our framework, we provide the pseudo code aligned with the data flow claimed above. 
```
# step 1
model = get_model(model_name)
scenario = build_scenario(scenario_name, ppl_cfg, **other_cfgs)
# step 2
instruction = build_instruction(instruction_type, prompt_cfg, icl_cfg, **other_cfgs)
inferencer = build_inferencer(inferencer_type, **other_cfgs)
metric = build_metric(scenario_name, **other_cfgs)

answers = inferencer.inference(scenario, model, instruction) # step 3 and 4

def inference(scenario, model, instruction):
    answers = []
    for sample in scenario:
        input = instruction.generate(sample) # step 3
        output = model.inference(input) # step 4
        answers += output
    return answers

def generate(sample):
    question = sample.get('question','') # step 3.1
    # step 3.2
    prompt = get_prompt(prompt_cfg) 
    input = prompt.format(question)
    # step 3.2.1 to 3.2.3
    ice = retriever.retrieve(sample)
    ice = [(image, prompt.format(text)) for (image, text) in ice]
    # input = [prompt_t.format(question) for prompt_t in multi_turn_prompt] # step 3.3
    input = (ice, sample['image'], input)
    return input

def inference(input):
    return model.direct_inference(input) # 4.1
    # return model.CoT_inference(input) # 4.2
    # return model.PPL_inference(input) # 4.3
    # return model.multiturn_inference(input) # 4.4

results = metric.metric(answers) # step 5
```

We sincerely hope these provide sufficient clarity. We will also provide a more detailed tutorial and toolkit in the code release version, and include these details in the final version of our paper after revision.

Reviewer comment (Novelty):
W1: Significance, Related work, and Execution. While I generally like the work that attempts to connect the dots and organize previous work, this work falls short. I do not think that the ChEF framework itself is a significant contribution as the 4 components of the Recipes are normally what people usually think about when it comes to evaluation. Thus, IMO, the main contributions lie in the instantiations of these Recipes or desiderata and their experimental results. However, the significance of this part is unclear due to two reasons.

Author response:
Besides the recipes, desiderata, and the experimental results, the ChEF evaluation framework itself is also a significant contribution. Creating a benchmarking framework is an immensely challenging endeavor. We have elucidated this in Section 2 of our overall response. Indeed, after extensive research on numerous MLLM benchmarks, as well as conducting in-depth studies on works that explore instruction tuning and training methods, it became quite natural for us to decompose the evaluation pipeline into the four modules that we proposed. However, **no existing work has introduced this concept before, and none have successfully interlinked these modules in a compatible and cohesive manner.** ChEF achieves this features, thereby encompassing a wide range of existing benchmarks and enabling a unified assessment of MLLMs. We claim that the development of ChEF as an evaluation framework is a main contribution of our work. We are glad that Reviewer mfBa and n4Xu shares this perspective, recognizing the significant strides we have made in advancing the field.

Reviewer comment (Novelty):
W1.1: First, it is unclear both in the main text and in the supplement how this work is better than existing work in terms of “scalability” and “comprehensiveness” (cf. the first paragraph of the intro). The paper has to put more emphasis on the discussion of related work in order for the reader to understand the significance.

Author response:
We introduce several related works in Section 1 of our overall response, and more related works in Appendix A. We also have outlined these comparisons with other works in Figure 1 (b) of our paper. Compared to the previous works, ChEF firstly decomposed the evaluation pipeline into four components. We also provide a toolkit for a quick extension of each component. 

- **More scalable**: We provided an example to demonstrate the scalability. Reviewer Ziwa mentioned the need for multi-image understanding task evaluations in the comments. We chose Winoground[1] dataset and spent only several hours to complete the construction of the recipe and the evaluation of multiple MLLMs. We simply build the recipe by adding the dataset to the scenario, defining prompts for this task, and then utilizing PPL along with a metric for calculating accuracy from ppl results. The modular design allows us to build the new recipe easily. However, this process of expanding the recipe is difficult to implement in any current benchmark that provides a single undecoupled evaluation pipeline.

- **More comprehensive**: ChEF has evaluated many traditional tasks and has also attempted to use more suitable methods for MLLM to evaluate detection tasks and fine-grained classification tasks, which have not been evaluated in any existing benchmarks. Additionally, ChEF has specially designed some unique recipes for evaluating the desiderata of MLLM, which have not been evaluated before. Based on these recipes, ChEF provides a comprehensive assessment of MLLMs.

Reviewer comment (Theory):
W1.2: Second, the desiderata in Section 2.3 themselves need more rationales/justification. Why do we care about these capabilities? Why do we instantiate them this way? For example, Hallucination consists of asking binary questions about the existence/absence of objects. Yet, this is not the only kind of hallucination. In general, it is unclear why the desiradata is what it is.

Author response:
We provide the rationales and justification of these desiderata in Section 3 of our overall response. More details of these desiderata and the rationale for their evaluation are supported in Appendix Section C. **The principle of selecting these desiderata are based on a survey and statistical analysis of the original LLM field, as well as the application of MLLM as an AI agent.** We have specified the recipes for these desiderata by conducting survey on MLLM and related works in NLP. For example, for hallucination, we use the method proposed in POPE[2]. Other methods also reference some approaches from NLP, adapted for multimodal tasks. Of course, these recipes are just attempts to evaluate these desiderata. We welcome the community to expand the evaluation of desiderata by constructing new and more reasonable recipes through the expansion of each module of ChEF.

Reviewer comment (Presentation):
W2: Clarity: related to W1, the paper would benefit from better presentation of desiradata. Perhaps having a table that lists down the 4 components. Justify why this is “versatile” evaluation.

Author response:
The desiderata are evlauated through specially designed recipes. We illustrate the recipes in Figure 3 in our paper, and show distinguished design of each desideratum in Figure 4 in the paper. For a clearer and more intuitive understanding of the recipes, we list a table to show the four components of the recipe for each desiderata in Appendix Section D.3, Table 8. We sincerely hope these provides sufficient clarity.