Repository: modelscope/evalscope
Issue: Macbook M3芯片执行Demo报错
Root cause: Configuration Error

## 问题描述 / Issue Description

Macbook M3芯片执行Demo报错：

RuntimeError: Placeholder storage has not been allocated on MPS device!


## 执行的代码或指令 / Code or Commands Executed

```bash
evalscope eval \                                                                                             
 --model Qwen/Qwen2.5-0.5B-Instruct \
 --datasets gsm8k arc \
 --limit 5
```

## 错误日志 / Error Log


```
Traceback (most recent call last):
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/bin/evalscope", line 8, in <module>
    sys.exit(run_cmd())
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/cli/cli.py", line 25, in run_cmd
    cmd.execute()
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/cli/start_eval.py", line 31, in execute
    run_task(self.args)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/run.py", line 31, in run_task
    return run_single_task(task_cfg, run_time)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/run.py", line 44, in run_single_task
    return evaluate_model(task_cfg, outputs)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/run.py", line 117, in evaluate_model
    res_dict = evaluator.eval()
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 368, in eval
    answers_list: list = self.get_answers(
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 176, in get_answers
    answer_ds: List[dict] = self._get_answer(
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 97, in _get_answer
    answer_ds: List[dict] = self.model_adapter.predict(inputs=input_prompts, infer_cfg=infer_cfg)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/models/chat_adapter.py", line 141, in predict
    responses = self._model_generate(queries, system_prompts, infer_cfg)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/evalscope/models/chat_adapter.py", line 109, in _model_generate
    output_ids = self.model.generate(**inputs, generation_config=self.generation_config)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/transformers/generation/utils.py", line 2223, in generate
    result = self._sample(
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/transformers/generation/utils.py", line 3211, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
    return func(*args, **kwargs)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 856, in forward
    outputs = self.model(
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 535, in forward
    inputs_embeds = self.embed_tokens(input_ids)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward
    return F.embedding(
  File "/Users/zhaozhixin/miniconda3/envs/evalscope/lib/python3.10/site-packages/torch/nn/functional.py", line 2551, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Placeholder storage has not been allocated on MPS device!
```

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
  - [ ] Windows
  - [* ] macOS
  - [ ] Ubuntu

- Python版本 / Python Version:
  - [ ] 3.11
  - [* ] 3.10
  - [ ] 3.9




尝试一下这样写呢：

```bash
evalscope eval \                                                                                             
 --model Qwen/Qwen2.5-0.5B-Instruct \
 --datasets gsm8k arc \
 --limit 5 \
 --model-args device_map=mps
``

> 尝试一下这样写呢：
> 
> evalscope eval \                                                                                             
>  --model Qwen/Qwen2.5-0.5B-Instruct \
>  --datasets gsm8k arc \
>  --limit 5 \
>  --model-args device_map=mps
> ``

感谢大佬，但是试了一下，还是不行

## 执行代码

`evalscope eval \
 --model Qwen/Qwen2.5-0.5B-Instruct \
 --model-args device_map=mps \
 --datasets gsm8k arc \
 --limit 5`

## 错误日志

`
RuntimeError: Placeholder storage has not been allocated on MPS device!
`

可以拉main分支代码安装，不设置 model-args 试试

从代码安装，然后改一下evalscope/models/local_model.py里的这一行就可以了
self.device = torch.device('cuda' if torch.cuda.is_available() else 'mps')

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！