Repository: open-compass/opencompass
Issue: [Bug] Srun上单机多卡测评报错
Root cause: Resource Mishandling

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

accelerate                1.4.0
torch                     2.1.0  

### Reproduces the problem - code/configuration sample

models = [
    dict(
        type=QwenSFTModel,
        abbr='step-8400-msq-1024',
        path='xxx/checkpoint-8400',
        max_out_len=1024,
        batch_size=1,
        run_cfg=dict(num_gpus=1),
    )
]


### Reproduces the problem - command or script

srun --gres=gpu:8 \
        --cpus-per-task 80 \
        --quotatype=auto \
    python run.py \
        --datasets ${task} \
        --models qwen_sft_v11  \
        --max-num-workers 8 \

### Reproduces the problem - error message

05/06 16:29:25 - OpenCompass - INFO - Task [qwen2.5-3b-sft-v1.1-step-8400-msq-1024/openai_humaneval_0]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
env//lib/python3.10/site-packages/accelerate/utils/modeling.py:1536: UserWarning: Current model requires 603984384 bytes of buffer for offloaded layers, which seems does not fit any GPU's remaining memory. If you are experiencing a OOM later, please consider using offload_buffers=True.
  warnings.warn(

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:08<00:08,  8.96s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:11<00:00,  5.05s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:11<00:00,  5.64s/it]
self.model.device=device(type='cpu')
05/06 16:29:52 - OpenCompass - INFO - using stop words: ['
', '
']
05/06 16:29:54 - OpenCompass - INFO - Try to load the data from /mnt/petrelfs/wangweiyun/.cache/opencompass/./data/humaneval/human-eval-v2-20210705.jsonl
05/06 16:29:54 - OpenCompass - INFO - Start inferencing [qwen2.5-3b-sft-v1.1-step-8400-msq-1024/openai_humaneval_0]
[2025-05-06 16:29:54,767] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2025-05-06 16:29:54,768] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...

  0%|          | 0/21 [00:00<?, ?it/s]messages=['
system\nYou are a helpful assistant.
\n
user\nRead the following function signature and docstring, and fully implement the function described. Your response should only contain the code for this function.\nfrom typing import List\n\n\ndef has_close_elements(numbers: List[float], threshold: float) -> bool:\n    """ Check if in given list of numbers, are any two numbers closer to each other than\n    given threshold.\n    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)\n    False\n    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)\n    True\n    """\n\n
\n
assistant\n']
05/06 16:29:54 - OpenCompass - INFO - Generation Args of Huggingface: 
05/06 16:29:54 - OpenCompass - INFO - {'stopping_criteria': [<opencompass.models.huggingface_above_v4_33._get_stopping_criteria.<locals>.MultiTokenEOSCriteria object at 0x7fb6452525f0>], 'max_new_tokens': 1024, 'pad_token_id': 151643}

  0%|          | 0/21 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "opencompass/tasks/openicl_infer.py", line 161, in <module>
    inferencer.run()
  File "opencompass/tasks/openicl_infer.py", line 89, in run
    self._inference()
  File "opencompass/tasks/openicl_infer.py", line 134, in _inference
    inferencer.inference(retriever,
  File "opencompass/openicl/icl_inferencer/icl_gen_inferencer.py", line 153, in inference
    results = self.model.generate_from_template(
  File "opencompass/models/base.py", line 201, in generate_from_template
    return self.generate(inputs, max_out_len=max_out_len, **kwargs)
  File "opencompass/models/huggingface_above_v4_33.py", line 1018, in generate
    outputs = self.model.generate(**tokens, **generation_kwargs)
  File "env//lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "env//lib/python3.10/site-packages/transformers/generation/utils.py", line 1479, in generate
    return self.greedy_search(
  File "env//lib/python3.10/site-packages/transformers/generation/utils.py", line 2340, in greedy_search
    outputs = self(
  File "env//lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "env//lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "env//lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 1173, in forward
    outputs = self.model(
  File "env//lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "env//lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "env//lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 1058, in forward
    layer_outputs = decoder_layer(
  File "env//lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "env//lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "env//lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 773, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "env//lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "env//lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "env//lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 260, in forward
    query_states = self.q_proj(hidden_states)
  File "env//lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "env//lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "env//lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'


### Other information

--max-num-workers 8时候就不会自动初始化到8张卡，在load_model里面print(f'{self.model.device=}')是CPU
但是我不使用--max-num-workers 8时候，用单卡是没问题的
我该如何来正确使用单机8卡来做模型并行呢