Repository: open-compass/VLMEvalKit
Issue: Error occurs when running Qwen2.5-VL-Instruct-7B with data-parallel inference via the transformers.
Root cause: External Dependency Breakage

I used the following command:

`CUDA_VISIBLE_DEVICE=0,1,2,3,4,5,6,7 torchrun --nproc-per-node=2 run.py --data HRBench4K  --model Qwen2.5-VL-7B-Instruct  --verbose`

Then it returns the following error:

```
[2025-06-22 06:00:27] ERROR - run.py: main - 481: Model Qwen2.5-VL-7B-Instruct x Dataset HRBench4K combination failed: shape '[1, 1303, 3584]' is invalid for input of size 2334976, skipping this combination.
Traceback (most recent call last):
  File "/remote-home/ywang/official_repo/VLMEvalKit/run.py", line 352, in main
    model = infer_data_job(
            ^^^^^^^^^^^^^^^
  File "/remote-home/ywang/official_repo/VLMEvalKit/vlmeval/inference.py", line 186, in infer_data_job
    model = infer_data(
            ^^^^^^^^^^^
  File "/remote-home/ywang/official_repo/VLMEvalKit/vlmeval/inference.py", line 149, in infer_data
    response = model.generate(message=struct, dataset=dataset_name)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/remote-home/ywang/official_repo/VLMEvalKit/vlmeval/vlm/base.py", line 116, in generate
    return self.generate_inner(message, dataset)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/remote-home/ywang/official_repo/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 621, in generate_inner
    return self.generate_inner_transformers(message, dataset=dataset)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/remote-home/ywang/official_repo/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 471, in generate_inner_transformers
    generated_ids = self.model.generate(
                    ^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/vlmevalkit/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/vlmevalkit/lib/python3.12/site-packages/transformers/generation/utils.py", line 2465, in generate
    result = self._sample(
             ^^^^^^^^^^^^^
  File "/root/miniconda3/envs/vlmevalkit/lib/python3.12/site-packages/transformers/generation/utils.py", line 3431, in _sample
    outputs = self(**model_inputs, return_dict=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/vlmevalkit/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/vlmevalkit/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/vlmevalkit/lib/python3.12/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1825, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/root/miniconda3/envs/vlmevalkit/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/vlmevalkit/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/vlmevalkit/lib/python3.12/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1171, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/vlmevalkit/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/vlmevalkit/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/vlmevalkit/lib/python3.12/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1033, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/vlmevalkit/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/vlmevalkit/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/vlmevalkit/lib/python3.12/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 964, in forward
    attn_output = attn_output.view(bsz, q_len, self.hidden_size)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: shape '[1, 1303, 3584]' is invalid for input of size 2334976
```

My transformers version is 4.51.3.