Repository: open-compass/VLMEvalKit
Issue: qwenvl2 run.py 无法一机多卡，每卡一个模型，并行推理一个测评
Root cause: Configuration Error

当我运行
```
torchrun --nproc-per-node=8 run.py --data DocVQA_TEST --model Qwen2-VL-2B-Instruct --verbose
```
出现以下错误
```
[{'role': 'user', 'content': [{'type': 'image', 'image': '/vlmeval/images/DocVQA_TEST/57348.jpg', 'min_pixels': 1003520, 'max_pixels': 12845056}, {'type': 'text', 'text': "What is the % of 'Providers of Capital' in the year 2010 based on 'Distribution of Value-Added' graph?\nPlease try to answer the question with short words or phrases if possible."}]}]
  0%|                             | 0/649 [00:04<?, ?it/s]
Traceback (most recent call last):
  File "/VLMEvalKit/run.py", line 226, in <module>
    main()
  File "/VLMEvalKit/run.py", line 140, in main
    model = infer_data_job(
  File "/VLMEvalKit/vlmeval/inference.py", line 164, in infer_data_job
    model = infer_data(
  File "/VLMEvalKit/vlmeval/inference.py", line 129, in infer_data
    response = model.generate(message=struct, dataset=dataset_name)
  File "/VLMEvalKit/vlmeval/vlm/base.py", line 115, in generate
    return self.generate_inner(message, dataset)
  File "/VLMEvalKit/vlmeval/vlm/qwen2_vl/model.py", line 100, in generate_inner
    generated_ids = self.model.generate(
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py", line 2053, in generate
    result = self._sample(
  File "/opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py", line 3003, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1686, in forward
    inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:3 and cuda:0! (when checking argument for argument mask in method wrapper_CUDA__masked_scatter_)
```
已有的issue似乎解决不了我的需求
我的模型够小，因此希望八卡八模型，每个卡推理一部分的文件
这在readme里似乎是 torchrun --nproc-per-node=8 run.py 来实现的

参考了现有 [issue#244](https://github.com/open-compass/VLMEvalKit/issues/224)
但qwen2vl的model.py里 模型load没有问题
```
 self.model = Qwen2VLForConditionalGeneration.from_pretrained(
            model_path, torch_dtype='auto', device_map='auto', attn_implementation='flash_attention_2'
        ).eval()
```
与issue 224里类似
```
self.model = AutoModel.from_pretrained(model_path, torch_dtype=torch.bfloat16,
                                       trust_remote_code=True,
                                       load_in_8bit=load_in_8bit, device_map='auto').eval()
```


device_map 自己做下制定，就可以解决啦

@M3Dade , 问题已修复

我也是用 device_map='auto' 加载的模型，在单卡就没问题。多卡报错：RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:0)在这一句上：input_ids = input_ids[attention_mask[i] == 1]