Repository: open-compass/VLMEvalKit
Issue: 评测76B模型在设置device map为auto后依然有报错
Root cause: Configuration Error

我评测的是本地模型，稍微改了一下代码，加载都是正常的。
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:29<00:00,  1.07it/s]
/mnt/code/yzy/VLMEvalKit/vlmeval/vlm/internvl_chat.py:165: UserWarning: Following kwargs received: {}, will use as generation config. 
  warnings.warn(f'Following kwargs received: {self.kwargs}, will use as generation config. ')
  0%|                                                                                                                                                                                            | 0/4329 [00:00<?, ?it/s]Generating with V2.0
Setting `pad_token_id` to `eos_token_id`:128003 for open-end generation.
  0%|                                                                                                                                                                                            | 0/4329 [00:03<?, ?it/s]
Traceback (most recent call last):
  File "/mnt/code/yzy/VLMEvalKit/run.py", line 208, in <module>
    main()
  File "/mnt/code/yzy/VLMEvalKit/run.py", line 122, in main
    model = infer_data_job(
  File "/mnt/code/yzy/VLMEvalKit/vlmeval/inference.py", line 151, in infer_data_job
    model = infer_data(
  File "/mnt/code/yzy/VLMEvalKit/vlmeval/inference.py", line 116, in infer_data
    response = model.generate(message=struct, dataset=dataset_name)
  File "/mnt/code/yzy/VLMEvalKit/vlmeval/vlm/base.py", line 115, in generate
    return self.generate_inner(message, dataset)
  File "/mnt/code/yzy/VLMEvalKit/vlmeval/vlm/internvl_chat.py", line 329, in generate_inner
    return self.generate_v2(message, dataset)
  File "/mnt/code/yzy/VLMEvalKit/vlmeval/vlm/internvl_chat.py", line 309, in generate_v2
    response = self.model.chat(
  File "/root/.cache/huggingface/modules/transformers_modules/InternVL2-Llama3-76B/modeling_internvl_chat.py", line 281, in chat
    generation_output = self.generate(
  File "/root/miniforge-pypy3/envs/VLMEval/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/InternVL2-Llama3-76B/modeling_internvl_chat.py", line 331, in generate
    outputs = self.language_model.generate(
  File "/root/miniforge-pypy3/envs/VLMEval/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/root/miniforge-pypy3/envs/VLMEval/lib/python3.10/site-packages/transformers/generation/utils.py", line 1989, in generate
    result = self._sample(
  File "/root/miniforge-pypy3/envs/VLMEval/lib/python3.10/site-packages/transformers/generation/utils.py", line 2975, in _sample
    next_tokens = next_tokens * unfinished_sequences + pad_token_id * (1 - unfinished_sequences)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!




我已经设置if listinstr(['InternVL2-Llama3-76B'], model_path):
            device_map = split_model(model_path.split('/')[-1])
            self.model = AutoModel.from_pretrained(
                model_path,
                torch_dtype=torch.bfloat16,
                load_in_8bit=load_in_8bit,
                trust_remote_code=True,
                low_cpu_mem_usage=True,
                device_map='auto').eval()
                #device_map=device_map).eval()

device_map 设置为 auto 无法运行，原始的代码使用 `python run.py --model InternVL2-76B --data RealWorldQA` 命令可以正常运行。

或可以改成使用 accelerate 进行 device_map 的构建：
```
from accelerate import init_empty_weights, infer_auto_device_map, dispatch_model
if listinstr(['InternVL2-Llama3-76B'], model_path):
    # device_map = split_model(model_path.split('/')[1])
    self.model = AutoModel.from_pretrained(
        model_path,
        torch_dtype=torch.bfloat16,
        load_in_8bit=load_in_8bit,
        trust_remote_code=True,
        low_cpu_mem_usage=True).eval()
    
    no_split_module = self.model._no_split_modules
    
    device_map = infer_auto_device_map(
        self.model,
        max_memory={
            0: '40GiB', 1: '40GiB', 2: '40GiB', 3: '40GiB'
        },
        no_split_module_classes=no_split_module)
    device_map['vision_model'] = 0
    device_map['mlp1'] = 0
    device_map['language_model.model.tok_embeddings'] = 0
    device_map['language_model.model.embed_tokens'] = 0
    device_map['language_model.output'] = 0
    device_map['language_model.model.norm'] = 0
    device_map['language_model.lm_head'] = 0
```

> device_map 设置为 auto 无法运行，原始的代码使用 `python run.py --model InternVL2-76B --data RealWorldQA` 命令可以正常运行。
> 
> 或可以改成使用 accelerate 进行 device_map 的构建：
> 
> ```
> from accelerate import init_empty_weights, infer_auto_device_map, dispatch_model
> if listinstr(['InternVL2-Llama3-76B'], model_path):
>     # device_map = split_model(model_path.split('/')[1])
>     self.model = AutoModel.from_pretrained(
>         model_path,
>         torch_dtype=torch.bfloat16,
>         load_in_8bit=load_in_8bit,
>         trust_remote_code=True,
>         low_cpu_mem_usage=True).eval()
>     
>     no_split_module = self.model._no_split_modules
>     
>     device_map = infer_auto_device_map(
>         self.model,
>         max_memory={
>             0: '40GiB', 1: '40GiB', 2: '40GiB', 3: '40GiB'
>         },
>         no_split_module_classes=no_split_module)
>     device_map['vision_model'] = 0
>     device_map['mlp1'] = 0
>     device_map['language_model.model.tok_embeddings'] = 0
>     device_map['language_model.model.embed_tokens'] = 0
>     device_map['language_model.output'] = 0
>     device_map['language_model.model.norm'] = 0
>     device_map['language_model.lm_head'] = 0
> ```

好的感谢我试一下

Thanks for the response from @junming-yang , plz feel free to re-open the issue if the problem still occurs @ZhiyuYUE