Repository: modelscope/evalscope
Issue: 如何使用NPU进行本地多卡方式调试
Root cause: Resource Mishandling

## 问题描述
在使用多卡NPU进行测评，例如32B模型需要两张卡，只能通过配置device_map=balanced进行内存分配。使用这种方式，当数据集过大时，会导致显存溢出。
观察显存使用量，32B平均分配，每张卡大约30G的显存占用，随着数据集的进一步评估，会增到50G、60G最终导致移除问题。

## EvalScope 版本 / Version (必须)
pip show evalscope
Name: evalscope
Version: 0.15.1

## 使用的工具 : Native 

## 执行的代码或指令 
nohup evalscope eval \
 --model ./models/DeepSeek-R1-Distill-Llama-8B \
 --generation-config do_sample=true,temperature=0.9,max_length=4096,max_new_tokens=1024,n=5\
 --model-args revision=master,precision=torch.float16,device_map=balanced \
 --work-dir ./output \
 --eval-batch-size 4 \
 --datasets gpqa live_code_bench aime24 math_500 \
 --dataset-args '{"aime24": {"local_path": "./data/aime24/data","metric_list":["Pass@1","Pass@5"]},"gpqa": {"local_path": "./data/gpqa","metric_list":["Pass@1","Pass@5"]},"live_code_bench": {"local_path": "./data/code_generation_lite","metric_list":["Pass@1","Pass@5"]},"math_500": {"local_path": "./data/math_500_ms","metric_list":["Pass@1","Pass@5"]}}' > ./log/0513/eval_ds_llama8b_0912.log 2>&1 &


## 错误日志 / Error Log

初始显存占用：
+------------------------------------------------------------------------------------------------+
| npu-smi 23.0.6.2                 Version: 23.0.6.2                                             |
+---------------------------+---------------+----------------------------------------------------+
| NPU   Name                | Health        | Power(W)    Temp(C)           Hugepages-Usage(page)|
| Chip                      | Bus-Id        | AICore(%)   Memory-Usage(MB)  HBM-Usage(MB)        |
+===========================+===============+====================================================+
| 0     910B2               | OK            | 91.7        32                0    / 0             |
| 0                         | 0000:C1:00.0  | 0           0    / 0          10234/ 65536         |
+===========================+===============+====================================================+
| 1     910B2               | OK            | 88.6        33                0    / 0             |
| 0                         | 0000:C2:00.0  | 0           0    / 0          11897/ 65536         |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
+===========================+===============+====================================================+
| 0       0                 | 176           | evalscope                | 6933                    |
+===========================+===============+====================================================+
| 1       0                 | 176           | evalscope                | 8597                    |
+===========================+===============+====================================================+

随着测评，显存占用：

+------------------------------------------------------------------------------------------------+
| npu-smi 23.0.6.2                 Version: 23.0.6.2                                             |
+---------------------------+---------------+----------------------------------------------------+
| NPU   Name                | Health        | Power(W)    Temp(C)           Hugepages-Usage(page)|
| Chip                      | Bus-Id        | AICore(%)   Memory-Usage(MB)  HBM-Usage(MB)        |
+===========================+===============+====================================================+
| 0     910B2               | OK            | 116.9       35                0    / 0             |
| 0                         | 0000:C1:00.0  | 0           0    / 0          55802/ 65536         |
+===========================+===============+====================================================+
| 1     910B2               | OK            | 122.1       38                0    / 0             |
| 0                         | 0000:C2:00.0  | 0           0    / 0          57507/ 65536         |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
+===========================+===============+====================================================+
| 0       0                 | 176           | evalscope                | 22501                   |
+===========================+===============+====================================================+
| 1       0                 | 176           | evalscope                | 24207                   |
+===========================+===============+====================================================+



RuntimeError: NPU out of memory. Tried to allocate 10.94 GiB (NPU 0; 60.97 GiB total capacity; 44.72 GiB already allocated; 44.72 GiB current active; 3.92 GiB free; 56.68 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.

## 运行环境 / Runtime Environment

- 操作系统 :Ubuntu

- Python版本 :3.10

目前NPU本地使用transformers推理确实观察到这个显存溢出问题，建议使用模型推理框架，例如 https://github.com/vllm-project/vllm-ascend ，拉起一个稳定的模型服务，然后指定API URL进行评测