Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Extremely Low Performance with LLaVA-1.5-7B on MM-Vet Benchmark
Root cause: Configuration Error

Hi, thanks for your great work. I am reproducing the evaluation results with the latest codebase and also the latest LLaVA codebase. The results of other benchmarks are matched or have minor differences. However, the performance with MM-Vet is very low. Could you please check the evaluation with MM-Vet from your side? Or could you please tell me what I should be care of?  Thank you!

|Tasks|Version|Filter|n-shot|    Metric    |Value |   |Stderr|
|-----|-------|------|-----:|--------------|-----:|---|------|
|mmvet|Yaml   |none  |     0|gpt_eval_score|1.3761|±  |N/A   |

I can also confirm that I get a low result using the command:
`lmms_eval --model llava --model_args pretrained="lmms-lab/llama3-llava-next-8b,conv_template=llava_llama_3" --tasks mmvet --batch_size 1 --log_samples --log_samples_suffix llava_next.mmvet --output_path ./logs/ `
I get:

|Tasks|Version|Filter|n-shot|    Metric    | Value |   |Stderr|
|-----|-------|------|-----:|--------------|------:|---|------|
|mmvet|Yaml   |none  |     0|gpt_eval_score|25.1376|±  |N/A   |

Which is significantly lower than the 37 and 37.6 you get with the Mistral and Vicuna versions.
This also seems to happen with the LLaVA-W benchmark:

|      Tasks      |Version|Filter|n-shot|        Metric        |Value|   |Stderr|
|-----------------|-------|------|-----:|----------------------|----:|---|------|
|llava_in_the_wild|Yaml   |none  |     0|gpt_eval_llava_conv   | 57.4|±  |N/A   |
|                 |       |none  |     0|gpt_eval_llava_detail | 80.8|±  |N/A   |
|                 |       |none  |     0|gpt_eval_llava_complex| 86.5|±  |N/A   |
|                 |       |none  |     0|gpt_eval_llava_all    | 76.5|±  |N/A   |

When the reported result [here](https://llava-vl.github.io/blog/2024-05-10-llava-next-stronger-llms/#3-llava-bench-wilder) is 80.1.

Hi @jeffhernandez1995, I run the same command you use instead I use multi processes.

### Command

```bash
accelerate launch --main_process_port 12345 --num_processes 8 -m lmms_eval --model llava --model_args pretrained="lmms-lab/llama3-llava-next-8b,conv_template=llava_llama_3,device_map=""" --tasks mmvet --batch_size 1 --log_samples --log_samples_suffix llava_next.mmvet --output_path ./logs/
```

and here are the results I get

<img width="950" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/92624596/7478477c-e899-4da4-9306-020bd1b1e68d">

Can you check whether in processing results, gpt parse every prediction correctly?


For llava in the wild, I have to say that gpt-4 sometimes may give different scores. Here is the original result logs we obtain when reporting mmvet and llava in the wild for the llama3_llava

[llava_in_the_wild.json](https://github.com/EvolvingLMMs-Lab/lmms-eval/files/15413573/llava_in_the_wild.json)
[mmvet.json](https://github.com/EvolvingLMMs-Lab/lmms-eval/files/15413574/mmvet.json)
[results.json](https://github.com/EvolvingLMMs-Lab/lmms-eval/files/15413575/results.json)




Hi @lambert-x , can you also check whether you process the result using gpt correctly? I can't reproduce using llava-1.5-7b. Also using the newest LLaVA-NeXT repo

Here is the result I get

<img width="797" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/92624596/f8c28d82-cfff-42ff-b331-8aaada73e47a">


I'll check with multiprocessing and the original LlaVA results and let you know.

I think the main reason should not caused by the multiprocessing. Possibly some errors occur such as failed to get gpt response during postprocess

Sorry, my bad. I, for some reason, changed the evaluator to `gpt-4-turbo` instead of the default one and must have forgotten. After reverting the change the scores are normal. Thank you for the great work you have put into this library!