Repository: open-compass/VLMEvalKit
Issue: Qwen2.5vl 7b can't reproduce Mathvista_MINI like leaderboard
Root cause: Configuration Error

**Results(66.6 not align with leaderboard 68.1):**
![Image](https://github.com/user-attachments/assets/25016051-8e02-41ec-a9df-5860ed9c5809)
**CUDA:**
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.04             Driver Version: 570.124.04     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100 80GB PCIe          Off |   00000001:00:00.0 Off |                    0 |
| N/A   33C    P0             41W /  300W |       6MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
**envs:** 
transformers：4.49.0
torch:2.5.1
torchvision:0.20.0

Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
accelerate                1.5.2                    pypi_0    pypi
aiofiles                  23.2.1                   pypi_0    pypi
annotated-types           0.7.0                    pypi_0    pypi
antlr4-python3-runtime    4.9.3                    pypi_0    pypi
anyio                     4.9.0                    pypi_0    pypi
av                        14.2.0                   pypi_0    pypi
bzip2                     1.0.8                h5eee18b_6  
ca-certificates           2025.2.25            h06a4308_0  
certifi                   2025.1.31                pypi_0    pypi
charset-normalizer        3.4.1                    pypi_0    pypi
click                     8.1.8                    pypi_0    pypi
contourpy                 1.3.1                    pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
decord                    0.6.0                    pypi_0    pypi
distro                    1.9.0                    pypi_0    pypi
einops                    0.8.1                    pypi_0    pypi
et-xmlfile                2.0.0                    pypi_0    pypi
exceptiongroup            1.2.2                    pypi_0    pypi
fastapi                   0.115.12                 pypi_0    pypi
ffmpy                     0.5.0                    pypi_0    pypi
filelock                  3.18.0                   pypi_0    pypi
flash-attn                2.7.4.post1              pypi_0    pypi
fonttools                 4.56.0                   pypi_0    pypi
fsspec                    2025.3.0                 pypi_0    pypi
gradio                    5.23.0                   pypi_0    pypi
gradio-client             1.8.0                    pypi_0    pypi
groovy                    0.1.2                    pypi_0    pypi
h11                       0.14.0                   pypi_0    pypi
httpcore                  1.0.7                    pypi_0    pypi
httpx                     0.28.1                   pypi_0    pypi
huggingface-hub           0.29.3                   pypi_0    pypi
idna                      3.10                     pypi_0    pypi
imageio                   2.37.0                   pypi_0    pypi
jinja2                    3.1.6                    pypi_0    pypi
jiter                     0.9.0                    pypi_0    pypi
kiwisolver                1.4.8                    pypi_0    pypi
ld_impl_linux-64          2.40                 h12ee557_0  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libstdcxx-ng              11.2.0               h1234567_1  
libuuid                   1.41.5               h5eee18b_0  
markdown-it-py            3.0.0                    pypi_0    pypi
markupsafe                3.0.2                    pypi_0    pypi
matplotlib                3.10.1                   pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0  
networkx                  3.4.2                    pypi_0    pypi
numpy                     2.2.4                    pypi_0    pypi
nvidia-cublas-cu12        12.4.5.8                 pypi_0    pypi
nvidia-cuda-cupti-cu12    12.4.127                 pypi_0    pypi
nvidia-cuda-nvrtc-cu12    12.4.127                 pypi_0    pypi
nvidia-cuda-runtime-cu12  12.4.127                 pypi_0    pypi
nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
nvidia-cufft-cu12         11.2.1.3                 pypi_0    pypi
nvidia-curand-cu12        10.3.5.147               pypi_0    pypi
nvidia-cusolver-cu12      11.6.1.9                 pypi_0    pypi
nvidia-cusparse-cu12      12.3.1.170               pypi_0    pypi
nvidia-cusparselt-cu12    0.6.2                    pypi_0    pypi
nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi
nvidia-nvtx-cu12          12.4.127                 pypi_0    pypi
omegaconf                 2.3.0                    pypi_0    pypi
openai                    1.68.2                   pypi_0    pypi
opencv-python             4.11.0.86                pypi_0    pypi
openpyxl                  3.1.5                    pypi_0    pypi
openssl                   1.1.1w               h7f8727e_0  
orjson                    3.10.16                  pypi_0    pypi
packaging                 24.2                     pypi_0    pypi
pandas                    2.2.3                    pypi_0    pypi
pillow                    11.1.0                   pypi_0    pypi
pip                       25.0            py310h06a4308_0  
portalocker               3.1.1                    pypi_0    pypi
protobuf                  6.30.1                   pypi_0    pypi
psutil                    7.0.0                    pypi_0    pypi
pydantic                  2.10.6                   pypi_0    pypi
pydantic-core             2.27.2                   pypi_0    pypi
pydub                     0.25.1                   pypi_0    pypi
pygments                  2.19.1                   pypi_0    pypi
pyparsing                 3.2.3                    pypi_0    pypi
python                    3.10.0               h12debd9_5  
python-dateutil           2.9.0.post0              pypi_0    pypi
python-dotenv             1.0.1                    pypi_0    pypi
python-multipart          0.0.20                   pypi_0    pypi
pytz                      2025.2                   pypi_0    pypi
pyyaml                    6.0.2                    pypi_0    pypi
qwen-vl-utils             0.0.10                   pypi_0    pypi
readline                  8.2                  h5eee18b_0  
regex                     2024.11.6                pypi_0    pypi
requests                  2.32.3                   pypi_0    pypi
rich                      13.9.4                   pypi_0    pypi
ruff                      0.11.2                   pypi_0    pypi
safehttpx                 0.1.6                    pypi_0    pypi
safetensors               0.5.3                    pypi_0    pypi
semantic-version          2.10.0                   pypi_0    pypi
sentencepiece             0.2.0                    pypi_0    pypi
setuptools                75.8.0          py310h06a4308_0  
shellingham               1.5.4                    pypi_0    pypi
six                       1.17.0                   pypi_0    pypi
sniffio                   1.3.1                    pypi_0    pypi
sqlite                    3.45.3               h5eee18b_0  
starlette                 0.46.1                   pypi_0    pypi
sty                       1.0.6                    pypi_0    pypi
sympy                     1.13.1                   pypi_0    pypi
tabulate                  0.9.0                    pypi_0    pypi
tiktoken                  0.9.0                    pypi_0    pypi
timeout-decorator         0.5.0                    pypi_0    pypi
tk                        8.6.14               h39e8969_0  
tokenizers                0.21.1                   pypi_0    pypi
tomlkit                   0.13.2                   pypi_0    pypi
torch                     2.5.1                    pypi_0    pypi
torchvision               0.20.0                   pypi_0    pypi
tqdm                      4.67.1                   pypi_0    pypi
transformers              4.49.0                   pypi_0    pypi
triton                    3.1.0                    pypi_0    pypi
typer                     0.15.2                   pypi_0    pypi
typing-extensions         4.12.2                   pypi_0    pypi
tzdata                    2025.2                   pypi_0    pypi
urllib3                   2.3.0                    pypi_0    pypi
uv                        0.6.9                    pypi_0    pypi
uvicorn                   0.34.0                   pypi_0    pypi
validators                0.34.0                   pypi_0    pypi
vlmeval                   0.1.0                     dev_0    <develop>
websockets                15.0.1                   pypi_0    pypi
wheel                     0.45.1          py310h06a4308_0  
xlsxwriter                3.2.2                    pypi_0    pypi
xz                        5.6.4                h5eee18b_1  
zlib                      1.2.13               h5eee18b_1  

Hi @alchemistyzz. The performance differences may be due to slight version discrepancies; you can pull the official image to try to reproduce the performance. 

@FangXinyu-0913 
Thanks for your reply  ,there's another question I'd like to raise.For example,I use swift x evalscope x VLMEvalKit for vllm acceleration,but as I dive into the repo,I find the temperature is set 0.0 by default,the eval result and infer response are not the same which are not deterministic.such as:

swift eval \
    --model Qwen/Qwen2.5-VL-7B-Instruct \
    --eval_backend VLMEvalKit \
    --infer_backend vllm \
    --eval_dataset MathVista_MINI \

[INFO:swift] args: DeployArguments(model='Qwen/Qwen2.5-VL-7B-Instruct', model_type='qwen2_5_vl', model_revision=None, task_type='causal_lm', torch_dtype=torch.bfloat16, attn_impl=None, num_labels=None, problem_type=None, rope_scaling=None, device_map=None, max_memory={}, local_repo_path=None, template='qwen2_5_vl', system=None, max_length=None, truncation_strategy='delete', max_pixels=None, tools_prompt='react_en', norm_bbox=None, response_prefix=None, padding_side='right', loss_scale='default', sequence_parallel_size=1, use_chat_template=True, template_backend='swift', dataset=[], val_dataset=[], split_dataset_ratio=0.01, data_seed=42, dataset_num_proc=1, streaming=False, enable_cache=False, download_mode='reuse_dataset_if_exists', columns={}, strict=False, remove_unused_columns=True, model_name=[None, None], model_author=[None, None], custom_dataset_info=[], quant_method=None, quant_bits=None, hqq_axis=None, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_type='nf4', bnb_4bit_use_double_quant=True, bnb_4bit_quant_storage=None, max_new_tokens=None, temperature=0.0, top_k=None, top_p=None, repetition_penalty=None, num_beams=1, stream=False, stop_words=[], logprobs=False, top_logprobs=None, ckpt_dir=None, load_dataset_config=None, lora_modules=[], tuner_backend='peft', train_type='lora', adapters=[], external_plugins=[], seed=42, model_kwargs={}, load_args=True, load_data_args=False, use_hf=True, hub_token=None, custom_register_path=[], ignore_args_error=False, use_swift_lora=False, tp=1, session_len=None, cache_max_entry_count=0.8, quant_policy=0, vision_batch_size=1, gpu_memory_utilization=0.9, tensor_parallel_size=1, pipeline_parallel_size=1, max_num_seqs=256, max_model_len=None, disable_custom_all_reduce=False, enforce_eager=False, limit_mm_per_prompt={}, vllm_max_lora_rank=16, enable_prefix_caching=False, merge_lora=False, safe_serialization=True, max_shard_size='5GB', infer_backend='vllm', result_path='/home/users/code/EvalBenchmark/result/Qwen2.5-VL-7B-Instruct/eval_result/20250327-134750.jsonl', metric=None, max_batch_size=1, ddp_backend=None, val_dataset_sample=None, host='0.0.0.0', port=8000, api_key=None, ssl_keyfile=None, ssl_certfile=None, owned_by='swift', served_model_name=None, verbose=False, log_interval=20, max_logprobs=20)

![Image](https://github.com/user-attachments/assets/bf10b2b1-5012-4270-b97e-c9f568ddcaec)
<html xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40">

## 1st RUN:

![Image](https://github.com/user-attachments/assets/c3805b4c-4560-46f1-b2db-99bd34645b53)

<head>

<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File
href="file:///C:/Users/V-YIZH~1/AppData/Local/Temp/msohtmlclip1/01/clip.htm">
<link rel=File-List
href="file:///C:/Users/V-YIZH~1/AppData/Local/Temp/msohtmlclip1/01/clip_filelist.xml">
<style>
<!--table
	{mso-displayed-decimal-separator:"\.";
	mso-displayed-thousand-separator:"\,";}
@page
	{margin:.75in .7in .75in .7in;
	mso-header-margin:.3in;
	mso-footer-margin:.3in;}
tr
	{mso-height-source:auto;
	mso-ruby-visibility:none;}
col
	{mso-width-source:auto;
	mso-ruby-visibility:none;}
br
	{mso-data-placement:same-cell;}
td
	{padding-top:1px;
	padding-right:1px;
	padding-left:1px;
	mso-ignore:padding;
	color:black;
	font-size:11.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:宋体;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-number-format:General;
	text-align:general;
	vertical-align:bottom;
	border:none;
	mso-background-source:auto;
	mso-pattern:auto;
	mso-protection:locked visible;
	white-space:nowrap;
	mso-rotate:0;}
ruby
	{ruby-align:left;}
rt
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:宋体;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-char-type:none;
	display:none;}
-->
</style>
</head>

<body link=blue vlink=purple>


Hint: Please answer   the question and provide the correct option letter, e.g., A, B, C, D, at the   end.     Question: Is the number of gray rubber objects in front of the small yellow   aeroplane greater than the number of big cyan matte fighters?     Choices:     (A) Yes     (B) No | ['Yes', 'No'] | Yes | multi_choice | text | 51 | general-vqa | synthetic scene | ['arithmetic reasoning'] | Super-CLEVR | visual question answering | A | A | A | Prefetch succeed | 1
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --



2nd RUN:

![Image](https://github.com/user-attachments/assets/85d1a078-a52d-4e2d-aeb8-119c3c095c96)
</body>

<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40">

<head>

<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File
href="file:///C:/Users/V-YIZH~1/AppData/Local/Temp/msohtmlclip1/01/clip.htm">
<link rel=File-List
href="file:///C:/Users/V-YIZH~1/AppData/Local/Temp/msohtmlclip1/01/clip_filelist.xml">
<style>
<!--table
	{mso-displayed-decimal-separator:"\.";
	mso-displayed-thousand-separator:"\,";}
@page
	{margin:.75in .7in .75in .7in;
	mso-header-margin:.3in;
	mso-footer-margin:.3in;}
.font5
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:宋体;
	mso-generic-font-family:auto;
	mso-font-charset:134;}
tr
	{mso-height-source:auto;
	mso-ruby-visibility:none;}
col
	{mso-width-source:auto;
	mso-ruby-visibility:none;}
br
	{mso-data-placement:same-cell;}
td
	{padding-top:1px;
	padding-right:1px;
	padding-left:1px;
	mso-ignore:padding;
	color:black;
	font-size:11.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:宋体;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-number-format:General;
	text-align:general;
	vertical-align:bottom;
	border:none;
	mso-background-source:auto;
	mso-pattern:auto;
	mso-protection:locked visible;
	white-space:nowrap;
	mso-rotate:0;}
.xl63
	{white-space:normal;}
ruby
	{ruby-align:left;}
rt
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:宋体;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-char-type:none;
	display:none;}
-->
</style>
</head>

<body link=blue vlink=purple>


Hint: Please answer   the question and provide the correct option letter, e.g., A, B, C, D, at the   end.     Question: Is the number of gray rubber objects in front of the small yellow   aeroplane greater than the number of big cyan matte fighters?     Choices:     (A) Yes     (B) No | ['Yes', 'No'] | Yes | multi_choice | text | 51 | general-vqa | synthetic scene | ['arithmetic reasoning'] | Super-CLEVR | visual question answering | A | To answer this question, let's   break it down step by step:          1. Identify the gray rubber objects in front of the small yellow   aeroplane:        - There is one gray rubber object   (an airplane) in front of the small yellow aeroplane.          2. Identify the big cyan matte fighters:        - There are no big cyan matte   fighters in the image.          3. Compare the quantities:        - The number of gray rubber   objects in front of the small yellow aeroplane is 1.        - The number of big cyan matte   fighters is 0.          Since 1 is greater than 0, the statement "Is the number of gray rubber   objects in front of the small yellow aeroplane greater than the number of big   cyan matte fighters?" is true.          The correct answer is: A | A | Prefetch succeed | 1
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --




</body>

</html>


 

Hi @alchemistyzz, The inference engine (like vllm, lmdeploy, and swift) uses techniques such as quantization to accelerate model inference, so even with a temperature of 0, there will be slight differences (in multiple inference cases), which is acceptable