Repository: open-compass/VLMEvalKit
Issue: torchrun多卡出现问题
Root cause: External Dependency Breakage

不使用torchrun --nproc-per-node=4，只单卡可以正常运行运行CUDA_VISIBLE_DEVICES=0 torchrun --nproc-per-node=1 run.py --data MMMath --model Qwen2.5-VL-7B-Instruct --verbose 
但是使用torchrun --nproc-per-node=4进行多卡推理会报错
CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc-per-node=4 run.py --data MMMath --model Qwen2.5-VL-7B-Instruct --verbose 
错误如下所示

W0503 09:01:07.540000 2338792 torch/distributed/run.py:792] 
W0503 09:01:07.540000 2338792 torch/distributed/run.py:792] *****************************************
W0503 09:01:07.540000 2338792 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0503 09:01:07.540000 2338792 torch/distributed/run.py:792] *****************************************
RANK: 0, WORLD_SIZE: 4, CUDA_VISIBLE_DEVICES: 0
RANK: 2, WORLD_SIZE: 4, CUDA_VISIBLE_DEVICES: 2
RANK: 1, WORLD_SIZE: 4, CUDA_VISIBLE_DEVICES: 1
RANK: 3, WORLD_SIZE: 4, CUDA_VISIBLE_DEVICES: 3

[2025-05-03 09:01:12] ERROR - misc.py: load_env - 219: Did not detect the .env file at /workspace/VLMEvalKit/.env, failed to load. 
[2025-05-03 09:01:12] ERROR - misc.py: load_env - 219: Did not detect the .env file at /workspace/VLMEvalKit/.env, failed to load. 
[2025-05-03 09:01:12,843] WARNING - RUN - run.py: main - 204: --reuse is not set, will not reuse previous (before one day) temporary files
[2025-05-03 09:01:12] WARNING - run.py: main - 204: --reuse is not set, will not reuse previous (before one day) temporary files
[2025-05-03 09:01:12] ERROR - misc.py: load_env - 219: Did not detect the .env file at /workspace/VLMEvalKit/.env, failed to load. 
[2025-05-03 09:01:12] ERROR - misc.py: load_env - 219: Did not detect the .env file at /workspace/VLMEvalKit/.env, failed to load. 
[2025-05-03 09:01:12] ERROR - misc.py: load_env - 219: Did not detect the .env file at /workspace/VLMEvalKit/.env, failed to load. 
[2025-05-03 09:01:12] ERROR - misc.py: load_env - 219: Did not detect the .env file at /workspace/VLMEvalKit/.env, failed to load. 
[2025-05-03 09:01:12] ERROR - misc.py: load_env - 219: Did not detect the .env file at /workspace/VLMEvalKit/.env, failed to load. 
[2025-05-03 09:01:12] ERROR - misc.py: load_env - 219: Did not detect the .env file at /workspace/VLMEvalKit/.env, failed to load. 
[rank0]:[W503 09:01:13.697599297 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 0]  using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
12f0e56ac84d:2338969:2338969 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
12f0e56ac84d:2338969:2338969 [0] NCCL INFO Bootstrap : Using eth0:172.17.0.2<0>
[rank2]:[W503 09:01:13.708968459 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 2]  using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank1]:[W503 09:01:13.737383913 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 1]  using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank3]:[W503 09:01:13.740090447 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 3]  using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
12f0e56ac84d:2338969:2338969 [0] NCCL INFO cudaDriverVersion 12010
NCCL version 2.21.5+cuda12.4
12f0e56ac84d:2338972:2338972 [0] NCCL INFO cudaDriverVersion 12010
12f0e56ac84d:2338972:2338972 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
12f0e56ac84d:2338972:2338972 [0] NCCL INFO Bootstrap : Using eth0:172.17.0.2<0>
12f0e56ac84d:2338969:2338969 [0] NCCL INFO Comm config Blocking set to 1
12f0e56ac84d:2338971:2338971 [0] NCCL INFO cudaDriverVersion 12010
12f0e56ac84d:2338971:2338971 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
12f0e56ac84d:2338970:2338970 [0] NCCL INFO cudaDriverVersion 12010
12f0e56ac84d:2338970:2338970 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
12f0e56ac84d:2338971:2338971 [0] NCCL INFO Bootstrap : Using eth0:172.17.0.2<0>
12f0e56ac84d:2338970:2338970 [0] NCCL INFO Bootstrap : Using eth0:172.17.0.2<0>
12f0e56ac84d:2338970:2338970 [0] NCCL INFO Comm config Blocking set to 1
12f0e56ac84d:2338972:2338972 [0] NCCL INFO Comm config Blocking set to 1
12f0e56ac84d:2338971:2338971 [0] NCCL INFO Comm config Blocking set to 1
12f0e56ac84d:2338969:2339155 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
12f0e56ac84d:2338969:2339155 [0] NCCL INFO P2P plugin IBext_v8
12f0e56ac84d:2338969:2339155 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
12f0e56ac84d:2338969:2339155 [0] NCCL INFO NET/IB : No device found.
12f0e56ac84d:2338969:2339155 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
12f0e56ac84d:2338969:2339155 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
12f0e56ac84d:2338969:2339155 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.2<0>
12f0e56ac84d:2338969:2339155 [0] NCCL INFO Using non-device net plugin version 0
12f0e56ac84d:2338969:2339155 [0] NCCL INFO Using network Socket
12f0e56ac84d:2338972:2339159 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
12f0e56ac84d:2338972:2339159 [0] NCCL INFO P2P plugin IBext_v8
12f0e56ac84d:2338972:2339159 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
12f0e56ac84d:2338972:2339159 [0] NCCL INFO NET/IB : No device found.
12f0e56ac84d:2338972:2339159 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
12f0e56ac84d:2338972:2339159 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
12f0e56ac84d:2338972:2339159 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.2<0>
12f0e56ac84d:2338972:2339159 [0] NCCL INFO Using non-device net plugin version 0
12f0e56ac84d:2338972:2339159 [0] NCCL INFO Using network Socket
12f0e56ac84d:2338971:2339161 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
12f0e56ac84d:2338971:2339161 [0] NCCL INFO P2P plugin IBext_v8
12f0e56ac84d:2338971:2339161 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
12f0e56ac84d:2338971:2339161 [0] NCCL INFO NET/IB : No device found.
12f0e56ac84d:2338971:2339161 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
12f0e56ac84d:2338971:2339161 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
12f0e56ac84d:2338971:2339161 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.2<0>
12f0e56ac84d:2338971:2339161 [0] NCCL INFO Using non-device net plugin version 0
12f0e56ac84d:2338971:2339161 [0] NCCL INFO Using network Socket
12f0e56ac84d:2338970:2339157 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
12f0e56ac84d:2338970:2339157 [0] NCCL INFO P2P plugin IBext_v8
12f0e56ac84d:2338970:2339157 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
12f0e56ac84d:2338970:2339157 [0] NCCL INFO NET/IB : No device found.
12f0e56ac84d:2338970:2339157 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
12f0e56ac84d:2338970:2339157 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
12f0e56ac84d:2338970:2339157 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.2<0>
12f0e56ac84d:2338970:2339157 [0] NCCL INFO Using non-device net plugin version 0
12f0e56ac84d:2338970:2339157 [0] NCCL INFO Using network Socket
12f0e56ac84d:2338969:2339155 [0] NCCL INFO ncclCommInitRank comm 0x559ee41b2b50 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId 18000 commId 0xfc552889eb5339ed - Init START
12f0e56ac84d:2338972:2339159 [0] NCCL INFO ncclCommInitRank comm 0x559ba373f870 rank 3 nranks 4 cudaDev 0 nvmlDev 3 busId af000 commId 0xfc552889eb5339ed - Init START
12f0e56ac84d:2338971:2339161 [0] NCCL INFO ncclCommInitRank comm 0x558a10886f70 rank 2 nranks 4 cudaDev 0 nvmlDev 2 busId 86000 commId 0xfc552889eb5339ed - Init START
12f0e56ac84d:2338970:2339157 [0] NCCL INFO ncclCommInitRank comm 0x563917771740 rank 1 nranks 4 cudaDev 0 nvmlDev 1 busId 3b000 commId 0xfc552889eb5339ed - Init START
12f0e56ac84d:2338972:2339159 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC
12f0e56ac84d:2338972:2339159 [0] NCCL INFO Setting affinity for GPU 3 to ffff,f00000ff,fff00000
12f0e56ac84d:2338972:2339159 [0] NCCL INFO NVLS multicast support is not available on dev 0
12f0e56ac84d:2338969:2339155 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC
12f0e56ac84d:2338969:2339155 [0] NCCL INFO Setting affinity for GPU 0 to 0fffff00,000fffff
12f0e56ac84d:2338969:2339155 [0] NCCL INFO NVLS multicast support is not available on dev 0
12f0e56ac84d:2338970:2339157 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC
12f0e56ac84d:2338970:2339157 [0] NCCL INFO Setting affinity for GPU 1 to 0fffff00,000fffff
12f0e56ac84d:2338970:2339157 [0] NCCL INFO NVLS multicast support is not available on dev 0
12f0e56ac84d:2338971:2339161 [0] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC
12f0e56ac84d:2338971:2339161 [0] NCCL INFO Setting affinity for GPU 2 to ffff,f00000ff,fff00000
12f0e56ac84d:2338971:2339161 [0] NCCL INFO NVLS multicast support is not available on dev 0
12f0e56ac84d:2338972:2339159 [0] NCCL INFO comm 0x559ba373f870 rank 3 nRanks 4 nNodes 1 localRanks 4 localRank 3 MNNVL 0
12f0e56ac84d:2338971:2339161 [0] NCCL INFO comm 0x558a10886f70 rank 2 nRanks 4 nNodes 1 localRanks 4 localRank 2 MNNVL 0
12f0e56ac84d:2338970:2339157 [0] NCCL INFO comm 0x563917771740 rank 1 nRanks 4 nNodes 1 localRanks 4 localRank 1 MNNVL 0
12f0e56ac84d:2338972:2339159 [0] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] -1/-1/-1->3->2
12f0e56ac84d:2338971:2339161 [0] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1
12f0e56ac84d:2338969:2339155 [0] NCCL INFO comm 0x559ee41b2b50 rank 0 nRanks 4 nNodes 1 localRanks 4 localRank 0 MNNVL 0
12f0e56ac84d:2338972:2339159 [0] NCCL INFO P2P Chunksize set to 131072
12f0e56ac84d:2338971:2339161 [0] NCCL INFO P2P Chunksize set to 131072
12f0e56ac84d:2338970:2339157 [0] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0
12f0e56ac84d:2338970:2339157 [0] NCCL INFO P2P Chunksize set to 131072
12f0e56ac84d:2338969:2339155 [0] NCCL INFO Channel 00/02 :    0   1   2   3
12f0e56ac84d:2338969:2339155 [0] NCCL INFO Channel 01/02 :    0   1   2   3
12f0e56ac84d:2338969:2339155 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
12f0e56ac84d:2338969:2339155 [0] NCCL INFO P2P Chunksize set to 131072
12f0e56ac84d:2338971:2339161 [0] NCCL INFO Channel 00 : 2[2] -> 3[3] via SHM/direct/direct
12f0e56ac84d:2338972:2339159 [0] NCCL INFO Channel 00 : 3[3] -> 0[0] via SHM/direct/direct
12f0e56ac84d:2338971:2339161 [0] NCCL INFO Channel 01 : 2[2] -> 3[3] via SHM/direct/direct
12f0e56ac84d:2338972:2339159 [0] NCCL INFO Channel 01 : 3[3] -> 0[0] via SHM/direct/direct
12f0e56ac84d:2338970:2339157 [0] NCCL INFO Channel 00 : 1[1] -> 2[2] via SHM/direct/direct
12f0e56ac84d:2338970:2339157 [0] NCCL INFO Channel 01 : 1[1] -> 2[2] via SHM/direct/direct
12f0e56ac84d:2338969:2339155 [0] NCCL INFO Channel 00 : 0[0] -> 1[1] via SHM/direct/direct
12f0e56ac84d:2338969:2339155 [0] NCCL INFO Channel 01 : 0[0] -> 1[1] via SHM/direct/direct
12f0e56ac84d:2338971:2339161 [0] NCCL INFO Connected all rings
12f0e56ac84d:2338970:2339157 [0] NCCL INFO Connected all rings
12f0e56ac84d:2338972:2339159 [0] NCCL INFO Connected all rings
12f0e56ac84d:2338969:2339155 [0] NCCL INFO Connected all rings
12f0e56ac84d:2338972:2339159 [0] NCCL INFO Channel 00 : 3[3] -> 2[2] via SHM/direct/direct
12f0e56ac84d:2338972:2339159 [0] NCCL INFO Channel 01 : 3[3] -> 2[2] via SHM/direct/direct
12f0e56ac84d:2338971:2339161 [0] NCCL INFO Channel 00 : 2[2] -> 1[1] via SHM/direct/direct
12f0e56ac84d:2338971:2339161 [0] NCCL INFO Channel 01 : 2[2] -> 1[1] via SHM/direct/direct
12f0e56ac84d:2338970:2339157 [0] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct/direct
12f0e56ac84d:2338970:2339157 [0] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct/direct
12f0e56ac84d:2338972:2339159 [0] NCCL INFO Connected all trees
12f0e56ac84d:2338972:2339159 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
12f0e56ac84d:2338972:2339159 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
12f0e56ac84d:2338971:2339161 [0] NCCL INFO Connected all trees
12f0e56ac84d:2338971:2339161 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
12f0e56ac84d:2338971:2339161 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
12f0e56ac84d:2338970:2339157 [0] NCCL INFO Connected all trees
12f0e56ac84d:2338970:2339157 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
12f0e56ac84d:2338970:2339157 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
12f0e56ac84d:2338969:2339155 [0] NCCL INFO Connected all trees
12f0e56ac84d:2338969:2339155 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
12f0e56ac84d:2338969:2339155 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
12f0e56ac84d:2338971:2339161 [0] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2, using internal tuner instead.
12f0e56ac84d:2338972:2339159 [0] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2, using internal tuner instead.
12f0e56ac84d:2338971:2339161 [0] NCCL INFO ncclCommInitRank comm 0x558a10886f70 rank 2 nranks 4 cudaDev 0 nvmlDev 2 busId 86000 commId 0xfc552889eb5339ed - Init COMPLETE
12f0e56ac84d:2338972:2339159 [0] NCCL INFO ncclCommInitRank comm 0x559ba373f870 rank 3 nranks 4 cudaDev 0 nvmlDev 3 busId af000 commId 0xfc552889eb5339ed - Init COMPLETE
12f0e56ac84d:2338970:2339157 [0] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2, using internal tuner instead.
12f0e56ac84d:2338969:2339155 [0] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2, using internal tuner instead.
12f0e56ac84d:2338970:2339157 [0] NCCL INFO ncclCommInitRank comm 0x563917771740 rank 1 nranks 4 cudaDev 0 nvmlDev 1 busId 3b000 commId 0xfc552889eb5339ed - Init COMPLETE
12f0e56ac84d:2338969:2339155 [0] NCCL INFO ncclCommInitRank comm 0x559ee41b2b50 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId 18000 commId 0xfc552889eb5339ed - Init COMPLETE
/workspace/VLMEvalKit/vlmeval/dataset/__init__.py:240: UserWarning: Dataset MMMath is not officially supported. 
  warnings.warn(f'Dataset {dataset_name} is not officially supported. ')
/workspace/VLMEvalKit/vlmeval/dataset/__init__.py:259: UserWarning: Will assume unsupported dataset MMMath as a Custom VQA dataset. 
  warnings.warn(f'Will assume unsupported dataset {dataset_name} as a Custom VQA dataset. ')
/workspace/VLMEvalKit/vlmeval/dataset/__init__.py:240: UserWarning: Dataset MMMath is not officially supported. 
  warnings.warn(f'Dataset {dataset_name} is not officially supported. ')
/workspace/VLMEvalKit/vlmeval/dataset/__init__.py:240: UserWarning: Dataset MMMath is not officially supported. 
  warnings.warn(f'Dataset {dataset_name} is not officially supported. ')
/workspace/VLMEvalKit/vlmeval/dataset/__init__.py:240: UserWarning: Dataset MMMath is not officially supported. 
  warnings.warn(f'Dataset {dataset_name} is not officially supported. ')
/workspace/VLMEvalKit/vlmeval/dataset/__init__.py:240: UserWarning: Dataset MMMath is not officially supported. 
  warnings.warn(f'Dataset {dataset_name} is not officially supported. ')
/workspace/VLMEvalKit/vlmeval/dataset/__init__.py:259: UserWarning: Will assume unsupported dataset MMMath as a Custom VQA dataset. 
  warnings.warn(f'Will assume unsupported dataset {dataset_name} as a Custom VQA dataset. ')
/workspace/VLMEvalKit/vlmeval/dataset/__init__.py:259: UserWarning: Will assume unsupported dataset MMMath as a Custom VQA dataset. 
  warnings.warn(f'Will assume unsupported dataset {dataset_name} as a Custom VQA dataset. ')
/workspace/VLMEvalKit/vlmeval/dataset/__init__.py:259: UserWarning: Will assume unsupported dataset MMMath as a Custom VQA dataset. 
  warnings.warn(f'Will assume unsupported dataset {dataset_name} as a Custom VQA dataset. ')
/workspace/VLMEvalKit/vlmeval/dataset/__init__.py:259: UserWarning: Will assume unsupported dataset MMMath as a Custom VQA dataset. 
  warnings.warn(f'Will assume unsupported dataset {dataset_name} as a Custom VQA dataset. ')
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.50, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.50, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.50, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.50, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour

Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour
You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour
You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour

Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:01<00:04,  1.08s/it]
Loading checkpoint shards:  20%|██        | 1/5 [00:01<00:04,  1.16s/it]
Loading checkpoint shards:  20%|██        | 1/5 [00:01<00:04,  1.17s/it]
Loading checkpoint shards:  20%|██        | 1/5 [00:01<00:04,  1.13s/it]
Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:02,  1.31it/s]
Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:02,  1.20it/s]
Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:02,  1.21it/s]
Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:02,  1.22it/s]
Loading checkpoint shards:  60%|██████    | 3/5 [00:02<00:01,  1.50it/s]
Loading checkpoint shards:  60%|██████    | 3/5 [00:02<00:01,  1.39it/s]
Loading checkpoint shards:  60%|██████    | 3/5 [00:02<00:01,  1.39it/s]
Loading checkpoint shards:  60%|██████    | 3/5 [00:02<00:01,  1.40it/s]
Loading checkpoint shards:  80%|████████  | 4/5 [00:02<00:00,  1.61it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.96it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.65it/s]

Infer Qwen2.5-VL-7B-Instruct/MMMath, Rank 1/4:   0%|          | 0/1475 [00:00<?, ?it/s]/workspace/VLMEvalKit/vlmeval/dataset/__init__.py:206: UserWarning: Dataset MMMath is a custom one and not annotated as `openended`, will treat as None. 
  warnings.warn(f'Dataset {dataset} is a custom one and not annotated as `openended`, will treat as {default}. ')  # noqa: E501

Loading checkpoint shards:  80%|████████  | 4/5 [00:02<00:00,  1.47it/s]
Loading checkpoint shards:  80%|████████  | 4/5 [00:02<00:00,  1.46it/s]
Loading checkpoint shards:  80%|████████  | 4/5 [00:02<00:00,  1.46it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.76it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.50it/s]

Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.77it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.51it/s]

Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.76it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:03<00:00,  1.50it/s]

Infer Qwen2.5-VL-7B-Instruct/MMMath, Rank 0/4:   0%|          | 0/1476 [00:00<?, ?it/s]/workspace/VLMEvalKit/vlmeval/dataset/__init__.py:206: UserWarning: Dataset MMMath is a custom one and not annotated as `openended`, will treat as None. 
  warnings.warn(f'Dataset {dataset} is a custom one and not annotated as `openended`, will treat as {default}. ')  # noqa: E501

Infer Qwen2.5-VL-7B-Instruct/MMMath, Rank 2/4:   0%|          | 0/1475 [00:00<?, ?it/s]/workspace/VLMEvalKit/vlmeval/dataset/__init__.py:206: UserWarning: Dataset MMMath is a custom one and not annotated as `openended`, will treat as None. 
  warnings.warn(f'Dataset {dataset} is a custom one and not annotated as `openended`, will treat as {default}. ')  # noqa: E501

Infer Qwen2.5-VL-7B-Instruct/MMMath, Rank 3/4:   0%|          | 0/1475 [00:00<?, ?it/s]/workspace/VLMEvalKit/vlmeval/dataset/__init__.py:206: UserWarning: Dataset MMMath is a custom one and not annotated as `openended`, will treat as None. 
  warnings.warn(f'Dataset {dataset} is a custom one and not annotated as `openended`, will treat as {default}. ')  # noqa: E501

Infer Qwen2.5-VL-7B-Instruct/MMMath, Rank 1/4:   0%|          | 0/1475 [00:02<?, ?it/s]
[2025-05-03 09:01:22,591] ERROR - RUN - run.py: main - 478: Model Qwen2.5-VL-7B-Instruct x Dataset MMMath combination failed: shape '[1, 1397, 3584]' is invalid for input of size 1251712, skipping this combination.
Traceback (most recent call last):
  File "/workspace/VLMEvalKit/run.py", line 350, in main
    model = infer_data_job(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 185, in infer_data_job
    model = infer_data(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 148, in infer_data
    response = model.generate(message=struct, dataset=dataset_name)
  File "/workspace/VLMEvalKit/vlmeval/vlm/base.py", line 116, in generate
    return self.generate_inner(message, dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 516, in generate_inner
    return self.generate_inner_transformers(message, dataset=dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 401, in generate_inner_transformers
    generated_ids = self.model.generate(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 2326, in generate
    result = self._sample(
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 3286, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1861, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1207, in forward
    layer_outputs = decoder_layer(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1069, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 907, in forward
    attn_output = attn_output.reshape(bsz, q_len, self.hidden_size).contiguous()
RuntimeError: shape '[1, 1397, 3584]' is invalid for input of size 1251712
[2025-05-03 09:01:22] ERROR - run.py: main - 478: Model Qwen2.5-VL-7B-Instruct x Dataset MMMath combination failed: shape '[1, 1397, 3584]' is invalid for input of size 1251712, skipping this combination.
Traceback (most recent call last):
  File "/workspace/VLMEvalKit/run.py", line 350, in main
    model = infer_data_job(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 185, in infer_data_job
    model = infer_data(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 148, in infer_data
    response = model.generate(message=struct, dataset=dataset_name)
  File "/workspace/VLMEvalKit/vlmeval/vlm/base.py", line 116, in generate
    return self.generate_inner(message, dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 516, in generate_inner
    return self.generate_inner_transformers(message, dataset=dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 401, in generate_inner_transformers
    generated_ids = self.model.generate(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 2326, in generate
    result = self._sample(
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 3286, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1861, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1207, in forward
    layer_outputs = decoder_layer(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1069, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 907, in forward
    attn_output = attn_output.reshape(bsz, q_len, self.hidden_size).contiguous()
RuntimeError: shape '[1, 1397, 3584]' is invalid for input of size 1251712

Infer Qwen2.5-VL-7B-Instruct/MMMath, Rank 0/4:   0%|          | 0/1476 [00:02<?, ?it/s]
[2025-05-03 09:01:23,020] ERROR - RUN - run.py: main - 478: Model Qwen2.5-VL-7B-Instruct x Dataset MMMath combination failed: shape '[1, 1926, 3584]' is invalid for input of size 1725696, skipping this combination.
Traceback (most recent call last):
  File "/workspace/VLMEvalKit/run.py", line 350, in main
    model = infer_data_job(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 185, in infer_data_job
    model = infer_data(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 148, in infer_data
    response = model.generate(message=struct, dataset=dataset_name)
  File "/workspace/VLMEvalKit/vlmeval/vlm/base.py", line 116, in generate
    return self.generate_inner(message, dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 516, in generate_inner
    return self.generate_inner_transformers(message, dataset=dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 401, in generate_inner_transformers
    generated_ids = self.model.generate(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 2326, in generate
    result = self._sample(
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 3286, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1861, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1207, in forward
    layer_outputs = decoder_layer(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1069, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 907, in forward
    attn_output = attn_output.reshape(bsz, q_len, self.hidden_size).contiguous()
RuntimeError: shape '[1, 1926, 3584]' is invalid for input of size 1725696
[2025-05-03 09:01:23] ERROR - run.py: main - 478: Model Qwen2.5-VL-7B-Instruct x Dataset MMMath combination failed: shape '[1, 1926, 3584]' is invalid for input of size 1725696, skipping this combination.
Traceback (most recent call last):
  File "/workspace/VLMEvalKit/run.py", line 350, in main
    model = infer_data_job(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 185, in infer_data_job
    model = infer_data(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 148, in infer_data
    response = model.generate(message=struct, dataset=dataset_name)
  File "/workspace/VLMEvalKit/vlmeval/vlm/base.py", line 116, in generate
    return self.generate_inner(message, dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 516, in generate_inner
    return self.generate_inner_transformers(message, dataset=dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 401, in generate_inner_transformers
    generated_ids = self.model.generate(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 2326, in generate
    result = self._sample(
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 3286, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1861, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1207, in forward
    layer_outputs = decoder_layer(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1069, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 907, in forward
    attn_output = attn_output.reshape(bsz, q_len, self.hidden_size).contiguous()
RuntimeError: shape '[1, 1926, 3584]' is invalid for input of size 1725696

Infer Qwen2.5-VL-7B-Instruct/MMMath, Rank 3/4:   0%|          | 0/1475 [00:02<?, ?it/s]
[2025-05-03 09:01:23,046] ERROR - RUN - run.py: main - 478: Model Qwen2.5-VL-7B-Instruct x Dataset MMMath combination failed: shape '[1, 1399, 3584]' is invalid for input of size 1253504, skipping this combination.
Traceback (most recent call last):
  File "/workspace/VLMEvalKit/run.py", line 350, in main
    model = infer_data_job(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 185, in infer_data_job
    model = infer_data(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 148, in infer_data
    response = model.generate(message=struct, dataset=dataset_name)
  File "/workspace/VLMEvalKit/vlmeval/vlm/base.py", line 116, in generate
    return self.generate_inner(message, dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 516, in generate_inner
    return self.generate_inner_transformers(message, dataset=dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 401, in generate_inner_transformers
    generated_ids = self.model.generate(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 2326, in generate
    result = self._sample(
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 3286, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1861, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1207, in forward
    layer_outputs = decoder_layer(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1069, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 907, in forward
    attn_output = attn_output.reshape(bsz, q_len, self.hidden_size).contiguous()
RuntimeError: shape '[1, 1399, 3584]' is invalid for input of size 1253504
[2025-05-03 09:01:23] ERROR - run.py: main - 478: Model Qwen2.5-VL-7B-Instruct x Dataset MMMath combination failed: shape '[1, 1399, 3584]' is invalid for input of size 1253504, skipping this combination.
Traceback (most recent call last):
  File "/workspace/VLMEvalKit/run.py", line 350, in main
    model = infer_data_job(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 185, in infer_data_job
    model = infer_data(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 148, in infer_data
    response = model.generate(message=struct, dataset=dataset_name)
  File "/workspace/VLMEvalKit/vlmeval/vlm/base.py", line 116, in generate
    return self.generate_inner(message, dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 516, in generate_inner
    return self.generate_inner_transformers(message, dataset=dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 401, in generate_inner_transformers
    generated_ids = self.model.generate(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 2326, in generate
    result = self._sample(
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 3286, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1861, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1207, in forward
    layer_outputs = decoder_layer(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1069, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 907, in forward
    attn_output = attn_output.reshape(bsz, q_len, self.hidden_size).contiguous()
RuntimeError: shape '[1, 1399, 3584]' is invalid for input of size 1253504

Infer Qwen2.5-VL-7B-Instruct/MMMath, Rank 2/4:   0%|          | 0/1475 [00:02<?, ?it/s]
[2025-05-03 09:01:23,048] ERROR - RUN - run.py: main - 478: Model Qwen2.5-VL-7B-Instruct x Dataset MMMath combination failed: shape '[1, 1416, 3584]' is invalid for input of size 1268736, skipping this combination.
Traceback (most recent call last):
  File "/workspace/VLMEvalKit/run.py", line 350, in main
    model = infer_data_job(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 185, in infer_data_job
    model = infer_data(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 148, in infer_data
    response = model.generate(message=struct, dataset=dataset_name)
  File "/workspace/VLMEvalKit/vlmeval/vlm/base.py", line 116, in generate
    return self.generate_inner(message, dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 516, in generate_inner
    return self.generate_inner_transformers(message, dataset=dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 401, in generate_inner_transformers
    generated_ids = self.model.generate(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 2326, in generate
    result = self._sample(
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 3286, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1861, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1207, in forward
    layer_outputs = decoder_layer(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1069, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 907, in forward
    attn_output = attn_output.reshape(bsz, q_len, self.hidden_size).contiguous()
RuntimeError: shape '[1, 1416, 3584]' is invalid for input of size 1268736
[2025-05-03 09:01:23] ERROR - run.py: main - 478: Model Qwen2.5-VL-7B-Instruct x Dataset MMMath combination failed: shape '[1, 1416, 3584]' is invalid for input of size 1268736, skipping this combination.
Traceback (most recent call last):
  File "/workspace/VLMEvalKit/run.py", line 350, in main
    model = infer_data_job(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 185, in infer_data_job
    model = infer_data(
  File "/workspace/VLMEvalKit/vlmeval/inference.py", line 148, in infer_data
    response = model.generate(message=struct, dataset=dataset_name)
  File "/workspace/VLMEvalKit/vlmeval/vlm/base.py", line 116, in generate
    return self.generate_inner(message, dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 516, in generate_inner
    return self.generate_inner_transformers(message, dataset=dataset)
  File "/workspace/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 401, in generate_inner_transformers
    generated_ids = self.model.generate(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 2326, in generate
    result = self._sample(
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 3286, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1861, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1207, in forward
    layer_outputs = decoder_layer(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1069, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 907, in forward
    attn_output = attn_output.reshape(bsz, q_len, self.hidden_size).contiguous()
RuntimeError: shape '[1, 1416, 3584]' is invalid for input of size 1268736
12f0e56ac84d:2338970:2338970 [0] NCCL INFO comm 0x563917771740 rank 1 nranks 4 cudaDev 0 busId 3b000 - Destroy COMPLETE
12f0e56ac84d:2338972:2338972 [0] NCCL INFO comm 0x559ba373f870 rank 3 nranks 4 cudaDev 0 busId af000 - Destroy COMPLETE
12f0e56ac84d:2338969:2338969 [0] NCCL INFO comm 0x559ee41b2b50 rank 0 nranks 4 cudaDev 0 busId 18000 - Destroy COMPLETE
12f0e56ac84d:2338971:2338971 [0] NCCL INFO comm 0x558a10886f70 rank 2 nranks 4 cudaDev 0 busId 86000 - Destroy COMPLETE


解决了这个问题！检查VLMEvalKit的requirement.txt环境要求后，我重新审视了报错原因，推测是transformer库版本导致的。我的版本为transformers-4.50.1，当我重装为transformers-4.49.0后可以正常工作了！
报错原因为： "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py"，RuntimeError: shape '[1, 1416, 3584]' is invalid for input of size 1268736。这里1416*3584/4（四卡）刚好=1268736，推测是新版本transformers的modeling_qwen2_5_vl.py代码修改了input处理逻辑？