Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Expected all tensors to be on the same device
Root cause: Resource Mishandling

srun -p ... \
    --job-name='internvl' \
    --gres=gpu:4 \
    --ntasks=1 \
    --cpus-per-task=12 \
    --kill-on-bad-exit=1 \
accelerate launch --num_processes 1 -m lmms_eval \
    --model=internvl2 \
    --model_args pretrained=OpenGVLab/InternVL2_5-38B,device_map=auto \
    --tasks=our_benchmark \
    --batch_size=1 \
    --log_samples \
    --log_samples_suffix=llava_v1.5_mme \
    --output_path=./logs/

**I encounter the load error when I use the above bash.**

error:

  File "/mnt/petrelfs/user/lmms-eval/lmms_eval/utils.py", line 533, in _wrapper
    return fn(*args, **kwargs)
  File "/mnt/petrelfs/user/lmms-eval/lmms_eval/evaluator.py", line 458, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)  # Choiszt run generate until
  File "/mnt/petrelfs/user/lmms-eval/lmms_eval/models/internvl2.py", line 315, in generate_until
    response, history = self.model.chat(self.tokenizer, pixel_values, contexts, gen_kwargs, num_patches_list=num_patches_list, history=None, return_history=True)
  File "/mnt/petrelfs/user/.cache/huggingface/modules/transformers_modules/InternVL2_5-38B/modeling_internvl_chat.py", line 290, in chat
    generation_output = self.generate(
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/mnt/petrelfs/user/.cache/huggingface/modules/transformers_modules/InternVL2_5-38B/modeling_internvl_chat.py", line 339, in generate
    outputs = self.language_model.generate(
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/transformers/generation/utils.py", line 2326, in generate
    result = self._sample(
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/transformers/generation/utils.py", line 3286, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
    return func(*args, **kwargs)
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 855, in forward
    outputs = self.model(
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 579, in forward
    layer_outputs = decoder_layer(
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/accelerate/hooks.py", line 176, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 257, in forward
    hidden_states = self.input_layernorm(hidden_states)
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/accelerate/hooks.py", line 176, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/mnt/petrelfs/user/anaconda3/envs/lmdeploy_qwen/lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 223, in forward
    return self.weight * hidden_states.to(input_dtype)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!


Hi Cuzyoung, did you resolve this issue? I am facing similar error.