Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1
Root cause: Algorithmic Error

Thanks for the benchmarks, 

I am trying to use multiple GPUs for evaluation with batch size 1 with following command - (Please note that I have long sequence and need to evaluate it in multiple GPUs)

CUDA_VISIBLE_DEVICES=0,1 accelerate launch --num_processes=1 -m lmms_eval --model internvl2   --model_args pretrained='OpenGVLab/InternVL2-8B,device_map=auto,trust_remote_code=True'   --tasks ours --batch_size 1 --log_samples --log_samples_suffix reproduce --output_path ./logs/

I can verify that model is sharded well in the two GPUs,  verified with lm.model.hf_device_map, output is -

{'language_model.model.layers.0': 0, 'language_model.model.layers.1': 0, 'language_model.model.layers.2': 0, 'language_model.model.layers.3': 0, 'language_model.model.layers.4': 0, 'language_model.model.layers.5': 0, 'language_model.model.layers.6': 0, 'language_model.model.layers.7': 0, 'language_model.model.layers.8': 0, 'language_model.model.layers.9': 0, 'language_model.model.layers.10': 0, 'language_model.model.layers.11': 1, 'language_model.model.layers.12': 1, 'language_model.model.layers.13': 1, 'language_model.model.layers.14': 1, 'language_model.model.layers.15': 1, 'language_model.model.layers.16': 1, 'language_model.model.layers.17': 1, 'language_model.model.layers.18': 1, 'language_model.model.layers.19': 1, 'language_model.model.layers.20': 1, 'language_model.model.layers.21': 1, 'language_model.model.layers.22': 1, 'language_model.model.layers.23': 1, 'language_model.model.layers.24': 1, 'language_model.model.layers.25': 1, 'language_model.model.layers.26': 1, 'language_model.model.layers.27': 1, 'language_model.model.layers.28': 1, 'language_model.model.layers.29': 1, 'language_model.model.layers.30': 1, 'language_model.model.layers.31': 0, 'language_model.model.layers.32': 1, 'vision_model': 0, 'mlp1': 0, 'language_model.model.tok_embeddings': 0, 'language_model.model.embed_tokens': 0, 'language_model.output': 0, 'language_model.model.norm': 0, 'language_model.lm_head': 0}

However, throws error as: Error during evaluation: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!. Please set `--verbosity=DEBUG` to get more information

This happens when model outputs from layer 10 are in cuda:0 whereas model wieghts of layer 11 are in cuda:1. 

I tried to use dispatch_model() from hugging_face for solving this, however it also doesnt work. 

Any help would be highly appreciated. 


Hello! Can you try not to use `accelerate` to launch? Thanks!

```sh
CUDA_VISIBLE_DEVICES=0,1 python -m lmms_eval --model internvl2 --model_args pretrained='OpenGVLab/InternVL2-8B,device_map=auto,trust_remote_code=True' --tasks ours --batch_size 1 --log_samples --log_samples_suffix reproduce --output_path ./logs/
```

Thanks for the prompt reply. 

Unfortunately, it doesnt work without accelerate. 

However, interestingly the default device_map works. The predefined device_map in - models/internvl2.py seems to be the issue. 

If we comment the line out -

device_map = split_model(pretrained.split("/")[-1], num_layers=num_layers)

And use device_map = 'auto' it works lika charm :) Maybe need to update the code.  

Hello! Thank you so much for the observation!!!

I remembered that this line is added to avoid device shading issue for InternVL2-26B, but it seems that it causes this issue on InternVL2-8B 😂 https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/481 The original code is from [here](https://huggingface.co/OpenGVLab/InternVL2-8B#multiple-gpus).

Maybe I need to test more model size to choose which models need to use this line of code. Thank you again!

Interesting!! Thanks for the heads up. Will use the device map provided for 26B variant.