Repository: open-compass/VLMEvalKit
Issue: Cannot Reproduce Results of LLaVA-1.5 on MMBench_DEV_CN
Root cause: Environment Incompatibility

Thank you for your awesome work!

I followed the README.md and used command `python run.py --data MMBench_DEV_CN --model llava_v1.5_13b --verbose`, and found  the overall accuracy is much lower than LLaVA-1.5's reported result (MMBench-cn 63.6)
<img width="307" alt="image" src="https://github.com/user-attachments/assets/5b227a41-c0da-41b7-afee-6c2693e962aa">



The overall accuracy reported in LLaVA-1.5 is based on the MMBench_CN dataset, aggregated from the TEST set and the  DEV set. In my already established environment, I tested the overall accuracy of MMBench_CN to match the official one and ested the accuracy of MMBench_DEV_CN as shown below. The fact that you measured low is probably related to the environment. The part of the environment I built is as follows, you can refer to the upgrade and re-test.

![image](https://github.com/user-attachments/assets/419bf3ec-a5f4-4f57-97c7-ee3a8e688da0)
![image](https://github.com/user-attachments/assets/6eee1b39-7e63-481b-a57e-e0e69c8faf72)


> The overall accuracy reported in LLaVA-1.5 is based on the MMBench_CN dataset, aggregated from the TEST set and the DEV set. In my already established environment, I tested the overall accuracy of MMBench_CN to match the official one and ested the accuracy of MMBench_DEV_CN as shown below. The fact that you measured low is probably related to the environment. The part of the environment I built is as follows, you can refer to the upgrade and re-test.
> 
> ![image](https://private-user-images.githubusercontent.com/81462011/362176693-419bf3ec-a5f4-4f57-97c7-ee3a8e688da0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUyMDUzNDksIm5iZiI6MTcyNTIwNTA0OSwicGF0aCI6Ii84MTQ2MjAxMS8zNjIxNzY2OTMtNDE5YmYzZWMtYTVmNC00ZjU3LTk3YzctZWUzYThlNjg4ZGEwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAxVDE1MzcyOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTFhOGIwZjliYzAyMmNjMTdjNzhlMjIzMzgzNjk3ZmE2MWMzYmExM2QxZDQ4OGIxMzhkODZlZmI4YWYzNmU0ZjImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.6YD_kVHYFgN8thWo6z24Yk0JqG1B9iK2yZ_la_60q7M) ![image](https://private-user-images.githubusercontent.com/81462011/362186068-6eee1b39-7e63-481b-a57e-e0e69c8faf72.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUyMDUzNDksIm5iZiI6MTcyNTIwNTA0OSwicGF0aCI6Ii84MTQ2MjAxMS8zNjIxODYwNjgtNmVlZTFiMzktN2U2My00ODFiLWE1N2UtZTBlNjljOGZhZjcyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAxVDE1MzcyOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTAzOWI2OWM5MzM4OWE1MGI4MTc5MjI1YjI4MzQyZGZjMzEwNDM0MmQ2MzdiODQ2ZTE2NGY2NTAxZGEwM2Q4NzgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.SLP7XK-ebM3Yvjm1H0ng0vhWJumX7Hw4dnvB0gJJN9E)

@FangXinyu-0913 Thank you very much for your reply. But your overall result of 54.72 is also lower than reported result (MMBench-cn 63.6) with a large margin. Do you have any idea about possible reasons?

Hi,

Please refer to [issue#439](https://github.com/open-compass/VLMEvalKit/issues/439#issuecomment-2328773836) to see if you are experiencing the same problem.

Best regards.


Hi, @ppalantir , 
We reproduce the problem and also find that the previous results of llava_v1.5 on some Chinese benchmarks (MMBench-CN, CCBench, etc.) can not be reproduced now. We have asked the authors of LLaVA but still cannot figure out the reason, so we have updated the leaderboard to align with the current evaluation results for now.