Repository: EvolvingLMMs-Lab/lmms-eval
Issue: N/A scores when running `mmbench`
Root cause: External Dependency Breakage

Hello, I am trying to evaluate the official Llava model on MMBench and am getting an N/A score after running:

```shell
accelerate launch --num_processes=8 -m lmms_eval --model llava   --model_args pretrained="liuhaotian/llava-v1.5-7b"   --tasks mmbench_en_dev  --batch_size 1 --log_samples --output_path ./logs/
```

Here is the output:

|    Tasks     |Version|Filter|n-shot|  Metric  |Value|   |Stderr|
|--------------|-------|------|-----:|----------|-----|---|------|
|mmbench_en_dev|Yaml   |none  |     0|submission|N/A  |±  |N/A   |

I think this is being caused by errors in the logs like:

```
04-04 00:13:20 [lmms_eval/models/llava.py:374] ERROR Error query and key must have the same dtype in generating
```

Is this benchmark expected to work as I ran it or is some other arg needed?

Actually this errors happens because of `use_flash_attention_2=True`, you may set it to false and see if things work normal?

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/70cc773aa8153e3ec58c9f5df73361b47ffe1a80/lmms_eval/models/llava.py#L75


> Actually this errors happens because of `use_flash_attention_2=True`, you may set it to false and see if things work normal?
> 
> https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/70cc773aa8153e3ec58c9f5df73361b47ffe1a80/lmms_eval/models/llava.py#L75

Another solution is to downgrade transformers to `4.36.2` or `4.37.2`.

We are looking for more detail feedback and community support to resolve the compatibility issue, we also observe some other minor issues between the `transformers`, `llava` and `lmms-eval` repo 🤷‍♀️.

Thanks for the tip re flash attention - setting it to false did remove the errors in the logs! However, I am still getting N/A in the scores in the outputs:

|      Tasks       |Version|Filter|n-shot|  Metric  |Value|   |Stderr|
|------------------|-------|------|-----:|----------|-----|---|------|
|mmbench_en        |N/A    |none  |     0|submission|N/A  |±  |N/A   |
| - mmbench_en_dev |Yaml   |none  |     0|submission|N/A  |±  |N/A   |
| - mmbench_en_test|Yaml   |none  |     0|submission|N/A  |±  |N/A   |

|  Groups  |Version|Filter|n-shot|  Metric  |Value|   |Stderr|
|----------|-------|------|-----:|----------|-----|---|------|
|mmbench_en|N/A    |none  |     0|submission|N/A  |±  |N/A   |

For reference I am attaching the results from running the command in the original post (perhaps it is now an issue with how the metrics are computed?)

[mmbench_en_dev_results.xlsx](https://github.com/EvolvingLMMs-Lab/lmms-eval/files/14863079/mmbench_en_dev_results.xlsx)
[results.json](https://github.com/EvolvingLMMs-Lab/lmms-eval/files/14863080/results.json)
[mmbench_en_dev.json](https://github.com/EvolvingLMMs-Lab/lmms-eval/files/14863081/mmbench_en_dev.json)


I am working on a PR to make the calculation of mmbench_dev in local. 

Previously we designed that it generates a submission file and calculate the score on MMBench official server.

@lewtun Hi could you help us to check if this PR could generate correct results for mmbencn_en_dev? You could checkout to this branch to see if it works well.

https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/46

> @lewtun Hi could you help us to check if this PR could generate correct results for mmbencn_en_dev? You could checkout to this branch to see if it works well.
> 
> #46

Sure, will test it now!

I can confirm the branch works as expected, and running

```shell
accelerate launch --num_processes=8 -m lmms_eval --model llava   --model_args pretrained="liuhaotian/llava-v1.5-7b"   --tasks mmbench_en_dev  --batch_size 1 --log_samples --output_path ./logs/
```

gives a score that looks consistent with the paper:

<img width="1291" alt="Screenshot 2024-04-05 at 10 49 31" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/26859204/554116cc-b8bf-46f5-8a7e-d15687107d9c">


|    Tasks     |Version|Filter|n-shot|    Metric    | Value |   |Stderr|
|--------------|-------|------|-----:|--------------|-------|---|------|
|mmbench_en_dev|Yaml   |none  |     0|gpt_eval_score|64.0893|±  |N/A   |
|              |       |none  |     0|submission    |N/A    |±  |N/A   |

In case it's relevant, here's the results over all variants of `mmbench`:

|      Tasks       |Version|Filter|n-shot|    Metric    | Value |   |Stderr |
|------------------|-------|------|-----:|--------------|-------|---|-------|
|mmbench           |N/A    |none  |     0|gpt_eval_score|50.0409|±  |54.2614|
|                  |       |none  |     0|submission    |N/A    |±  |N/A    |
| - mmbench_cn_cc  |Yaml   |none  |     0|gpt_eval_score|23.5294|±  |N/A    |
|                  |       |none  |     0|submission    |N/A    |±  |N/A    |
| - mmbench_cn_dev |Yaml   |none  |     0|gpt_eval_score|55.6701|±  |N/A    |
|                  |       |none  |     0|submission    |N/A    |±  |N/A    |
| - mmbench_cn_test|Yaml   |none  |     0|submission    |N/A    |±  |N/A    |
| - mmbench_en_dev |Yaml   |none  |     0|gpt_eval_score|64.0893|±  |N/A    |
|                  |       |none  |     0|submission    |N/A    |±  |N/A    |
| - mmbench_en_test|Yaml   |none  |     0|submission    |N/A    |±  |N/A    |

Is it expected that the test splits don't have a `gpt_eval_score`?

Hello! Yes, for the test split, you have to submit the generated file to the [MMBench site](https://sso.openxlab.org.cn/authentication?redirect=https://mmbench.opencompass.org.cn/mmbench-submission&clientId=bvdgdz4zjwr7o1r2wo3y&lang=en-US) to get the result, since in the dataset given, the `answer` column is masked so we cannot calculate the result locally.

Thanks, will close this issue since it is now resolved