Repository: modelscope/evalscope
Issue: 循环调用报错了。
Root cause: Resource Mishandling

## 问题描述 / Issue Description

请简要描述您遇到的问题。 / Please briefly describe the issue you encountered.

## 使用的工具 / Tools Used
- [✔] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena  /竞技场模式


## 执行的代码或指令 / Code or Commands Executed

请提供您执行的主要代码或指令。 / Please provide the main code or commands you executed. 例如 / For example:

```python
# 例如：执行的Python代码 / Python code executed
import json
import os
import re
from tqdm import tqdm
from evalscope.run import run_task
from evalscope.config import TaskConfig

json_path = "./lora_path.json"

root_dir = "/workspace/experiment_log/train"

adapters_paths = []

with open(json_path, "r") as f:
    adapters_paths = json.load(f)


for adapter in tqdm(adapters_paths):
    paths = os.listdir(os.path.join(root_dir, adapter))
    # 筛选出，带 merged的 path
    merged_paths = [path for path in paths if "merged" in path]

    if not merged_paths:
        print(f"{adapter} 没有找到 merged_path")
        continue
    checkpoint_path = merged_paths[0]
    checkpoint_path = os.path.join(root_dir, adapter, checkpoint_path)
    task_cfg = TaskConfig(
        model=checkpoint_path,
        datasets=['general_qa'],  # 数据格式，选择题格式固定为 'general_qa'
        dataset_args={
            'general_qa': {
                "local_path": "/workspace/test_data",  # 自定义数据集路径
                    "subset_list": [
                        "rsh_train_data_600"       # 评测数据集名称，上述 *.jsonl 中的 *
                    ]
                }
            },
        
    )


    run_task(task_cfg=task_cfg)
```

## 错误日志 / Error Log

写了一个这样子的循环，每次循环到第二次的时候，上一次的显存并没有释放，导致报错。请问大佬们是怎么解决的呢？

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
  - [ ] Windows
  - [ ] macOS
  - [✔ ] Ubuntu

- Python版本 / Python Version:
  - [ ] 3.11
  - [✔] 3.10
  - [ ] 3.9


## 其他信息 / Additional Information

如果有其他相关信息，请在此处提供。 / If there is any other relevant information, please provide it here.


请问，具体报什么错，有相关log吗

> 请问，具体报什么错，有相关log吗

 50%|██████████████████████████████████████████████████▌                                                  | 1/2 [1:07:44<1:07:44, 4064.87s/it]2025-03-21 11:46:46,668 - evalscope - INFO - Args: Task config is provided with TaskConfig type.
Loading checkpoint shards:  21%|████████████████▉                                                              | 3/14 [00:19<01:12,  6.55s/it]
 50%|██████████████████████████████████████████████████▌                                                  | 1/2 [1:08:05<1:08:05, 4085.01s/it]
Traceback (most recent call last):
  File "/workspace/experiment_script/eval/eval_args_3_20.py", line 43, in <module>
    run_task(task_cfg=task_cfg)
  File "/usr/local/lib/python3.10/site-packages/evalscope/run.py", line 31, in run_task
    return run_single_task(task_cfg, run_time)
  File "/usr/local/lib/python3.10/site-packages/evalscope/run.py", line 44, in run_single_task
    return evaluate_model(task_cfg, outputs)
  File "/usr/local/lib/python3.10/site-packages/evalscope/run.py", line 106, in evaluate_model
    base_model = get_local_model(task_cfg)
  File "/usr/local/lib/python3.10/site-packages/evalscope/models/local_model.py", line 73, in get_local_model
    base_model = LocalModel(
  File "/usr/local/lib/python3.10/site-packages/evalscope/models/local_model.py", line 45, in __init__
    self.model = AutoModelForCausalLM.from_pretrained(
  File "/usr/local/lib/python3.10/site-packages/modelscope/utils/hf_util/patcher.py", line 225, in from_pretrained
    module_obj = module_class.from_pretrained(
  File "/usr/local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/usr/local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4264, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/usr/local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4777, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
  File "/usr/local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 942, in _load_state_dict_into_meta_model
    set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs)
  File "/usr/local/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 330, in set_module_tensor_to_device
    new_value = value.to(device)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 270.00 MiB. GPU 0 has a total capacity of 79.14 GiB of which 178.75 MiB is free. Process 16162 has 78.96 GiB memory in use. Of the allocated memory 78.46 GiB is allocated by PyTorch, and 3.13 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables

第一次运行完的 task 没有释放显存

可以用main分支代码再试一下


> 可以用main分支代码再试一下

您好一样的报错

![Image](https://github.com/user-attachments/assets/b85a7fcb-5886-48ed-add3-147bf5e943e0)

我用下面的代码测试过，显存正常释放了没有累积，你尝试在每个循环结束时手动调用`torch.cuda.empty_cache()` 和`gc.collect()`呢：

```python
from evalscope.config import TaskConfig

task_cfg1 = TaskConfig(
    model='Qwen/Qwen2.5-0.5B-Instruct',
    model_id='model1',
    datasets=['iquiz'],
    limit=10
)
task_cfg2 = TaskConfig(
    model='Qwen/Qwen2.5-0.5B-Instruct',
    model_id='model2',
    datasets=['iquiz'],
    limit=10
)
task_cfg3 = TaskConfig(
    model='Qwen/Qwen2.5-0.5B-Instruct',
    model_id='model3',
    datasets=['iquiz'],
    limit=10
)

run_task(task_cfg=[task_cfg1, task_cfg2, task_cfg3])
```

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！