Repository: open-compass/VLMEvalKit
Issue: BUG on loading  qwen2.5-vl-3b model.
Root cause: Validation Gap

**I used the MS Swift framework to train qwen2.5-vl-3b with grpo in the Lora mode, and then used the following Swift export script to export the model. Then I ran the following VLMEvalKit script and got an error: (Before train, loading is OK, so i don't know what's wrong!!!).** 
```
[2025-06-06 18:05:28] ERROR - RUN - run.py: main - 494: Model Qwen2.5-VL-3B-Instruct x Dataset DocVQA_TEST combination failed: Trying to set a tensor of shape torch.Size([2048]) in "bias" (which has shape torch.Size([1280])), this looks incorrect., skipping this combination.
Traceback (most recent call last):
  File "/home/admin/workspace/aop_lab/20241011/7851/Repository_25/VLMEvalKit/run.py", line 366, in main
    model = infer_data_job(
  File "/home/admin/workspace/aop_lab/20241011/7851/Repository_25/VLMEvalKit/vlmeval/inference.py", line 185, in infer_data_job
    model = infer_data(
  File "/home/admin/workspace/aop_lab/20241011/7851/Repository_25/VLMEvalKit/vlmeval/inference.py", line 117, in infer_data
    model = supported_VLM[model_name](**kwargs) if isinstance(model, str) else model
  File "/home/admin/workspace/aop_lab/20241011/7851/Repository_25/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 303, in __init__
    self.model = MODEL_CLS.from_pretrained(
  File "/opt/conda/envs/vlmtoolkit/lib/python3.10/site-packages/transformers/modeling_utils.py", line 262, in _wrapper
    return func(*args, **kwargs)
  File "/opt/conda/envs/vlmtoolkit/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4319, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/opt/conda/envs/vlmtoolkit/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4897, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
  File "/opt/conda/envs/vlmtoolkit/lib/python3.10/site-packages/transformers/modeling_utils.py", line 896, in _load_state_dict_into_meta_model
    set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs)
  File "/opt/conda/envs/vlmtoolkit/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 292, in set_module_tensor_to_device
    raise ValueError(
ValueError: Trying to set a tensor of shape torch.Size([2048]) in "bias" (which has shape torch.Size([1280])), this looks incorrect.
```
**swift export script**
```
swift export \
--ckpt_dir '/home/admin/workspace/aop_lab/Repository_25/ms-swift-m/output_orz_math/v3-20250604-030852/checkpoint-10000' \
--merge_lora true \
```
**VLMEvalKit script**
```
torchrun --nproc-per-node=4 run.py --data DocVQA_TEST --model Qwen2.5-VL-3B-Instruct --verbose  --mode infer
```

**But I use the following to load directly using the transformers library, there is no problem.**
```
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "/home/admin/workspace/aop_lab/20241011/7851/Repository_25/ms-swift-m/output_orz_math/v3-20250604-030852/checkpoint-10000-merged", 
    torch_dtype="auto", 
    device_map="auto"
)
min_pixels = 256*28*28
max_pixels = 1280*28*28
processor = AutoProcessor.from_pretrained("/home/admin/workspace/aop_lab/20241011/7851/Repository_25/ms-swift-m/output_orz_math/v3-20250604-030852/checkpoint-10000-merged", min_pixels=min_pixels, max_pixels=max_pixels)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)]
output_text = processor.batch_decode(generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False)
```

**My Env:**
flash_attn               2.7.4.post1
torch                    2.6.0+cu124
transformers             4.49.0

Before train: Both VLMEvalKit and Transformers are OK. 
After train and export, Transformers is OK, but VLMEvalKit gets wrong.

Is there some gaps between VLMEvalKit and Transformers ?

I encountered the same issue. Have you resolved it?

我解决这个问题了，vlmevalkit里面需要通过模型路径的名称来确定需要加载的模型类型，因为路径里面没有带2.5，默认加载qwen2vl了，所以会报错。
```
if listinstr(['2.5', '2_5', 'qwen25'], model_path.lower()):
            from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
            MODEL_CLS = Qwen2_5_VLForConditionalGeneration
            self.processor = AutoProcessor.from_pretrained(model_path)
        else:
            from transformers import Qwen2VLForConditionalGeneration, Qwen2VLProcessor
            MODEL_CLS = Qwen2VLForConditionalGeneration
            self.processor = Qwen2VLProcessor.from_pretrained(model_path)
```
在路径里面补充上对应的词就行了

> 我解决这个问题了，vlmevalkit里面需要通过模型路径的名称来确定需要加载的模型类型，因为路径里面没有带2.5，默认加载qwen2vl了，所以会报错。
> 
> ```
> if listinstr(['2.5', '2_5', 'qwen25'], model_path.lower()):
>             from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
>             MODEL_CLS = Qwen2_5_VLForConditionalGeneration
>             self.processor = AutoProcessor.from_pretrained(model_path)
>         else:
>             from transformers import Qwen2VLForConditionalGeneration, Qwen2VLProcessor
>             MODEL_CLS = Qwen2VLForConditionalGeneration
>             self.processor = Qwen2VLProcessor.from_pretrained(model_path)
> ```
> 
> 在路径里面补充上对应的词就行了

你好请问你最后成功的版本transformer是什么版本的，我发现我低版本的时候qwen2.5会报错，版本高了qwen2又会报错

Qwen2.5-Vl官方推荐版本为`pip install transformers==4.51.3`

For evaluation, I can run both Qwen2 & Qwen2.5 VL w. transformers 4.51.0 

> > 我解决这个问题了，vlmevalkit里面需要通过模型路径的名称来确定需要加载的模型类型，因为路径里面没有带2.5，默认加载qwen2vl了，所以会报错。
> > ```
> > if listinstr(['2.5', '2_5', 'qwen25'], model_path.lower()):
> >             from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
> >             MODEL_CLS = Qwen2_5_VLForConditionalGeneration
> >             self.processor = AutoProcessor.from_pretrained(model_path)
> >         else:
> >             from transformers import Qwen2VLForConditionalGeneration, Qwen2VLProcessor
> >             MODEL_CLS = Qwen2VLForConditionalGeneration
> >             self.processor = Qwen2VLProcessor.from_pretrained(model_path)
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > 在路径里面补充上对应的词就行了
> 
> 你好请问你最后成功的版本transformer是什么版本的，我发现我低版本的时候qwen2.5会报错，版本高了qwen2又会报错



> 我解决这个问题了，vlmevalkit里面需要通过模型路径的名称来确定需要加载的模型类型，因为路径里面没有带2.5，默认加载qwen2vl了，所以会报错。
> 
> ```
> if listinstr(['2.5', '2_5', 'qwen25'], model_path.lower()):
>             from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
>             MODEL_CLS = Qwen2_5_VLForConditionalGeneration
>             self.processor = AutoProcessor.from_pretrained(model_path)
>         else:
>             from transformers import Qwen2VLForConditionalGeneration, Qwen2VLProcessor
>             MODEL_CLS = Qwen2VLForConditionalGeneration
>             self.processor = Qwen2VLProcessor.from_pretrained(model_path)
> ```
> 
> 在路径里面补充上对应的词就行了

写这个代码的员工真应该开除。。。。。