Repository: open-compass/VLMEvalKit
Issue: Different evaluate result between different VLMEvalKit version
Root cause: Algorithmic Error

The official LLaVA-v1.5-7B model gets 1362 points in MME Perception scores under the newest code 69b7b5e.
![image](https://github.com/user-attachments/assets/2897aeb6-6464-4a65-a4cb-af91b86a8cc1)
But it turns out to be 1497 under the 027e38c8df0c312135da70a2abf68fe706b05629 commit.
![51a97534b233e0574351b8ef2318272](https://github.com/user-attachments/assets/1e99e9ac-2904-4573-a9d3-ce32a2397dde)
I was sure about the consistent LLaVA code between this two experiement, why would this happened?


I've check that once again. The problem still exists.

i also notice this problem, however, when i check the code in https://github.com/open-compass/VLMEvalKit/commit/69b7b5ed9d8fcdc2b8ac3f48e9974a38111a92b1. I found that it only modified three parts：
1. Modify various log records, such as warning to logging
2. Change single quotes to double quotes
3. Modify some code problems
Have you solved this problem?

> i also notice this problem, however, when i check the code in [69b7b5e](https://github.com/open-compass/VLMEvalKit/commit/69b7b5ed9d8fcdc2b8ac3f48e9974a38111a92b1). I found that it only modified three parts：
> 
> 1. Modify various log records, such as warning to logging
> 2. Change single quotes to double quotes
> 3. Modify some code problems
>    Have you solved this problem?

No. I just directly use the https://github.com/open-compass/VLMEvalKit/commit/027e38c8df0c312135da70a2abf68fe706b05629 version lol.

I can confirm the issue on commit 9f21ee8.
`"perception","reasoning","OCR","artwork","celebrity","code_reasoning","color","commonsense_reasoning","count","existence","landmark","numerical_calculation","position","posters","scene","text_translation"
"1342.2684073629453","301.42857142857144","130.0","112.0","125.88235294117646","62.5","151.66666666666669","106.42857142857143","93.33333333333333","185.0","138.5","40.0","115.0","140.1360544217687","150.75","92.5"
`
The counting score is 93.3.
The counting score is 155 in a reference experiment (https://github.com/haotian-liu/LLaVA/issues/927).
The score of counting is extremely low as Llava-1.5 tends to report the actual count instead of answering yes or no.
![image](https://github.com/user-attachments/assets/04266fdf-29f0-45b1-b78c-90a1cd8df66c)


The same phenomenon is observed on MME-RealWorld-Lite, the counting score is 0.

This might be related to the system prompt or the instruction following capability of the model. But I am not sure how to address this.

Can @kennymckormick maybe have a look? Thanks!

> I can confirm the issue on commit [9f21ee8](https://github.com/open-compass/VLMEvalKit/commit/9f21ee8a8251f735d241e7009d507b6bbedcd1b2). `"perception","reasoning","OCR","artwork","celebrity","code_reasoning","color","commonsense_reasoning","count","existence","landmark","numerical_calculation","position","posters","scene","text_translation" "1342.2684073629453","301.42857142857144","130.0","112.0","125.88235294117646","62.5","151.66666666666669","106.42857142857143","93.33333333333333","185.0","138.5","40.0","115.0","140.1360544217687","150.75","92.5" ` The counting score is 93.3. The counting score is 155 in a reference experiment ([haotian-liu/LLaVA#927](https://github.com/haotian-liu/LLaVA/issues/927)). The score of counting is extremely low as Llava-1.5 tends to report the actual count instead of answering yes or no. ![image](https://private-user-images.githubusercontent.com/108799629/391930401-04266fdf-29f0-45b1-b78c-90a1cd8df66c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzMyNDkzMDQsIm5iZiI6MTczMzI0OTAwNCwicGF0aCI6Ii8xMDg3OTk2MjkvMzkxOTMwNDAxLTA0MjY2ZmRmLTI5ZjAtNDViMS1iNzhjLTkwYTFjZDhkZjY2Yy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQxMjAzJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MTIwM1QxODAzMjRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kNWRmZDNiZGI0OTllNzVlMGM0MjNiMDdjNWFmYjE0OTM2MjAzMDM0NmIxNWM0MGFjYzM0NDc2YjNkNjU0YjczJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9._u_qgweOpxfsZaQzzrvJtn1ymBOKll6_1vt1NaZNTSo)
> 
> The same phenomenon is observed on MME-RealWorld-Lite, the counting score is 0.
> 
> This might be related to the system prompt or the instruction following capability of the model. But I am not sure how to address this.
> 
> Can @kennymckormick maybe have a look? Thanks!

Good job! Thanks for the testing. I'm trying to figure out the problem, too.

Anyone have solved it?

Anyone have solved it?

Anyone have solved it?

Anyone have solved it?