Repository: modelscope/evalscope
Issue: evalscope Live_code_bench review阶段卡住
Root cause: Resource Mishandling

## 问题描述 / Issue Description

请简要描述您遇到的问题。 / Please briefly describe the issue you encountered.
使用evalscope live_code_bench测试代码生成，在review阶段卡住

## EvalScope 版本 / Version (必须)
v0.13.1

## 使用的工具 / Tools Used
- [x] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena  /竞技场模式


## 执行的代码或指令 / Code or Commands Executed

请提供您执行的主要代码或指令。 / Please provide the main code or commands you executed. 例如 / For example:

```python
from evalscope import TaskConfig, run_task
from evalscope.constants import EvalType

task_cfg = TaskConfig(
    model='deepseek-r1',   
    api_url='http://xxx/v1',
    api_key='EMPTY',
    eval_type=EvalType.SERVICE,
    datasets=[
        'live_code_bench',
    ],
    dataset_args={
        'live_code_bench': {
            'subset_list': ['v5']
        }
    },
    eval_batch_size=20,       
    generation_config={       
        'max_tokens': 16384,  
        'temperature': 0.6,   
        'top_p': 0.95,        
        'n': 1                
    },
    stream=True               
)

run_task(task_cfg=task_cfg)
```

```bash
# 例如：在终端中执行的指令 / Terminal command executed
nohup sudo  python3 run.py > nohup.out 2>&1 &
```

## 错误日志 / Error Log

请粘贴完整的错误日志或控制台输出。 / Please paste the full error log or console output. 例如 / For example:

```
2025-04-01 10:20:46,395 - evalscope - INFO - **** Start evaluating on dataset AI-ModelScope/code_generation_lite ****
2025-04-01 10:20:46,395 - evalscope - INFO - Loading dataset from hub: AI-ModelScope/code_generation_lite
2025-04-01 10:20:46,552 - evalscope - INFO - Loading dataset: dataset_name: AI-ModelScope/code_generation_lite > subsets: ['v5']
2025-04-01 10:20:46,552 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from code_generation_lite. Please make sure that you can trust the external codes.
2025-04-01 10:20:46,862 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from AI-ModelScope/code_generation_lite. Please make sure that you can trust the external codes.
2025-04-01 10:20:46,862 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from AI-ModelScope/code_generation_lite. Please make sure that you can trust the external codes.
2025-04-01 10:20:46,862 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from AI-ModelScope/code_generation_lite. Please make sure that you can trust the external codes.
Transforming data: 100%|██████████| 167/167 [00:11<00:00, 14.19it/s]
2025-04-01 10:21:06,138 - evalscope - INFO - Use default settings: > few_shot_num: 0, > few_shot_split: None, > target_eval_split: test
Predicting(v5): 100%|██████████| 167/167 [3:18:56<00:00, 71.48s/it]
2025-04-01 13:40:03,022 - evalscope - INFO - Dump predictions to ./outputs/20250401_102045/predictions/DeepSeek-R1/live_code_bench_v5.jsonl.
Reviewing(v5):  29%|██▊       | 48/167 [00:12<00:26,  4.46it/s]
```

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
  - [ ] Windows
  - [ ] macOS
  - [ ] Ubuntu
  - [x] CentOS7

- Python版本 / Python Version:
  - [x] 3.11
  - [ ] 3.10
  - [ ] 3.9


## 其他信息 / Additional Information

如果有其他相关信息，请在此处提供。 / If there is any other relevant information, please provide it here.

我尝试了两次都是在review阶段卡住，两次卡住的通过率不同，第一次50%


参考： #402 
设置`judge_worker_num=1`

> 参考： [#402](https://github.com/modelscope/evalscope/issues/402) 设置`judge_worker_num=1`

根据文档说明，默认就是1。https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html#judge
--judge-worker-num: 裁判模型并发数，默认为1

文档是最新的版本的说明，v0.13.1 版本默认值是8，现在改成1了

> 参考： [#402](https://github.com/modelscope/evalscope/issues/402) 设置`judge_worker_num=1`

在开头的python代码上加上了judge_worker_num=1后重跑了下，但是还是卡住了。快四个小时的时间，review一直停留在21/167。请问有方法来排查或者解决这个问题？
```python
    datasets=[
        'live_code_bench',
    ],
    judge_worker_num=1,
    dataset_args={
        'live_code_bench': {
            'subset_list': ['v5']
        }
    },
```

使用v0.13.2也会有这个没问题吗，如果再卡住的话可以用py-spy看一下程序卡在哪里

请问下  问题解决了吗？ v0.13.2遇到同样的问题，设置judge_worker_num=1 也会一直在review阶段卡住

@wwp1988 可以用main分支代码再试一下，测试部分代码替换成qwen官方实现了 https://github.com/QwenLM/QwQ/blob/main/eval/eval/livecodebench_v5_utils/testing_util.py ，不过我这边还是复现不了judge_worker_num=1 卡住的问题

已经试了main分支，还是会卡住，这个是 py-spy dump信息

Thread 87 (idle): "MainThread"
    wait (threading.py:320)
    wait (threading.py:607)
    as_completed (concurrent/futures/_base.py:245)
    __iter__ (tqdm/std.py:1181)
    get_reviews (evalscope/evaluator/evaluator.py:299)
    eval (evalscope/evaluator/evaluator.py:415)
    evaluate_model (evalscope/run.py:126)
    run_single_task (evalscope/run.py:44)
    run_task (evalscope/run.py:31)
    <module> (run_eval_code_new.py:26)
Thread 236 (idle): "Thread-1"
    wait (threading.py:324)
    wait (threading.py:607)
    run (tqdm/_monitor.py:60)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 238 (idle): "Thread-2"
    wait (threading.py:324)
    wait (threading.py:607)
    run (tqdm/_monitor.py:60)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 239 (idle): "ThreadPoolExecutor-2_0"
    wait (threading.py:320)
    wait (threading.py:607)
    as_completed (concurrent/futures/_base.py:245)
    evaluate_generations (evalscope/benchmarks/live_code_bench/evaluate_utils.py:134)
    codegen_metrics (evalscope/benchmarks/live_code_bench/evaluate_utils.py:166)
    match (evalscope/benchmarks/live_code_bench/live_code_bench_adapter.py:81)
    _get_review (evalscope/evaluator/evaluator.py:229)
    process_single_review (evalscope/evaluator/evaluator.py:290)
    run (concurrent/futures/thread.py:58)
    _worker (concurrent/futures/thread.py:83)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 2704 (idle): "Thread-119"
    select (selectors.py:416)
    wait (multiprocessing/connection.py:931)
    wait_result_broken_or_wakeup (concurrent/futures/process.py:385)
    run (concurrent/futures/process.py:320)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 2705 (idle): "QueueFeederThread"
    wait (threading.py:320)
    _feed (multiprocessing/queues.py:231)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)


@wwp1988 请问你使用的系统是什么，windows吗

不是，Ubuntu系统
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"


> 请问下 问题解决了吗？ v0.13.2遇到同样的问题，设置judge_worker_num=1 也会一直在review阶段卡住

没有，未找到解决方法

再拉一下main分支代码，支持了debug参数：
```
dataset_args={
  'live_code_bench': {
      'subset_list': ['v4_v5'],
      'extra_params': {
          'start_date': '2024-12-01',
          'end_date': '2025-02-28',
          'debug': False,
      },
},
```
如果再卡住的话，可以打断点看一下，我这边也没找到卡住的原因 :(


最新main分支代码，直接测试不会卡住了  给力~！！ @Yunnglin 

使用 v0.14.0依然会卡住。两个进程的dump数据如下
```
Process 13907: python3 runtest.py
Python v3.11.10 (/data/python/bin/python3.11)

Thread 13907 (idle): "ThreadPoolExecutor-2_0"
    _flush_std_streams (multiprocessing/util.py:442)
    __init__ (multiprocessing/popen_fork.py:16)
    _Popen (multiprocessing/context.py:281)
    start (multiprocessing/process.py:121)
    start (multiprocessing/managers.py:563)
    Manager (multiprocessing/context.py:57)
    codegen_check_correctness (evalscope/benchmarks/live_code_bench/evaluate_utils.py:26)
    evaluate_generations_by_problem (evalscope/benchmarks/live_code_bench/evaluate_utils.py:65)
    _process_worker (concurrent/futures/process.py:261)
    run (multiprocessing/process.py:108)
    _bootstrap (multiprocessing/process.py:314)
    _launch (multiprocessing/popen_fork.py:71)
    __init__ (multiprocessing/popen_fork.py:19)
    _Popen (multiprocessing/context.py:281)
    start (multiprocessing/process.py:121)
    _spawn_process (concurrent/futures/process.py:785)
    _launch_processes (concurrent/futures/process.py:775)
    _start_executor_manager_thread (concurrent/futures/process.py:748)
    submit (concurrent/futures/process.py:809)
    <dictcomp> (evalscope/benchmarks/live_code_bench/evaluate_utils.py:128)
    evaluate_generations (evalscope/benchmarks/live_code_bench/evaluate_utils.py:127)
    codegen_metrics (evalscope/benchmarks/live_code_bench/evaluate_utils.py:166)
    match (evalscope/benchmarks/live_code_bench/live_code_bench_adapter.py:81)
    _get_review (evalscope/evaluator/evaluator.py:229)
    process_single_review (evalscope/evaluator/evaluator.py:290)
    run (concurrent/futures/thread.py:58)
    _worker (concurrent/futures/thread.py:83)
    run (threading.py:982)
    _bootstrap_inner (threading.py:1045)
    _bootstrap (threading.py:1002)
```

```
Process 27288: python3 runtest.py
Python v3.11.10 (/data/python/bin/python3.11)

Thread 27288 (idle): "MainThread"
    wait (threading.py:327)
    wait (threading.py:629)
    as_completed (concurrent/futures/_base.py:243)
    __iter__ (tqdm/std.py:1181)
    get_reviews (evalscope/evaluator/evaluator.py:299)
    eval (evalscope/evaluator/evaluator.py:415)
    evaluate_model (evalscope/run.py:126)
    run_single_task (evalscope/run.py:44)
    run_task (evalscope/run.py:31)
    <module> (runtest.py:28)
Thread 27879 (idle): "Thread-1"
    wait (threading.py:331)
    wait (threading.py:629)
    run (tqdm/_monitor.py:60)
    _bootstrap_inner (threading.py:1045)
    _bootstrap (threading.py:1002)
Thread 27911 (idle): "Thread-2"
    wait (threading.py:331)
    wait (threading.py:629)
    run (tqdm/_monitor.py:60)
    _bootstrap_inner (threading.py:1045)
    _bootstrap (threading.py:1002)
Thread 8959 (idle): "ThreadPoolExecutor-2_0"
    wait (threading.py:327)
    wait (threading.py:629)
    as_completed (concurrent/futures/_base.py:243)
    evaluate_generations (evalscope/benchmarks/live_code_bench/evaluate_utils.py:134)
    codegen_metrics (evalscope/benchmarks/live_code_bench/evaluate_utils.py:166)
    match (evalscope/benchmarks/live_code_bench/live_code_bench_adapter.py:81)
    _get_review (evalscope/evaluator/evaluator.py:229)
    process_single_review (evalscope/evaluator/evaluator.py:290)
    run (concurrent/futures/thread.py:58)
    _worker (concurrent/futures/thread.py:83)
    run (threading.py:982)
    _bootstrap_inner (threading.py:1045)
    _bootstrap (threading.py:1002)
Thread 13908 (idle): "Thread-23"
    select (selectors.py:415)
    wait (multiprocessing/connection.py:948)
    wait_result_broken_or_wakeup (concurrent/futures/process.py:417)
    run (concurrent/futures/process.py:344)
    _bootstrap_inner (threading.py:1045)
    _bootstrap (threading.py:1002)
Thread 13909 (idle): "QueueFeederThread"
    wait (threading.py:327)
    _feed (multiprocessing/queues.py:231)
    run (threading.py:982)
    _bootstrap_inner (threading.py:1045)
    _bootstrap (threading.py:1002)
```

@moluzhui 用main分支代码试试

@Yunnglin main分支[源码安装](https://github.com/modelscope/evalscope?tab=readme-ov-file#method-2-install-from-source)后可以了，请问哪个release会修复（以便在其他服务器上通过pip安装）

@moluzhui 最新的pip包（v0.15.1）已经修复了

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！

@Yunnglin 

请问为什么open-compass使用`ProcessPoolExecutor`，能正常跑。而这边却是会hang？
https://github.com/open-compass/opencompass/blob/190cc0c/opencompass/datasets/livecodebench/evaluator.py#L146

难道是evaluator `get_reviews`又套了一层`ThreadPoolExecutor`？
https://github.com/modelscope/evalscope/blob/52d0110d/evalscope/evaluator/evaluator.py#L244

如果不能用ProcessPoolExecutor多进程的话，感觉LiveCodeBench的reviews会很慢？

@dynamicheart 这个可能也不是`ThreadPoolExecutor`嵌套`multiprocessing`的问题，同样的写法，humaneval没有这个问题。不过马上会加上sandbox功能，代码就不需要额外的`multiprocessing`来运行了。

@Yunnglin 先记录下：

1. `ProcessPoolExecutor`已经全部submit，正在等结果的as_completed返回：https://github.com/modelscope/evalscope/blob/v0.14.0/evalscope/benchmarks/live_code_bench/evaluate_utils.py#L134
2. `ProcessPoolExecutor`的`_ExecutorManagerThread`等待工作进程的结果返回(resultQueue)、进程正常或则异常退出信息(worker_sentinels)、submit事件唤醒(wakeup_reader)，结果是wait没有返回：https://github.com/python/cpython/blob/3.10/Lib/concurrent/futures/process.py#L385
3. 其中resultQueue是一个SimpleQueue，需要用Pickler序列化和管道传输：https://github.com/python/cpython/blob/3.10/Lib/multiprocessing/queues.py#L369

首先，去除掉ProcessPoolExecutor，所有case，无论成功还是失败的，都能正常结束，说明execute这套逻辑没有问题；其次，worker_sentinels这个信号也没有被触发，说明工作进程没有异常退出。

因此，我大概率怀疑hang是可能是管道传输有些问题，或者Pickler有些问题，毕竟livecodebench可能某些case数据量也比较大。具体可能还得py-spy看下子进程那边hang在哪里。

Pipe的一些关键代码：
1. https://github.com/python/cpython/blob/3.10/Lib/multiprocessing/connection.py#L365-L388
2. https://github.com/python/cpython/blob/3.10/Lib/concurrent/futures/process.py#L392
3. https://github.com/python/cpython/blob/3.10/Lib/multiprocessing/queues.py#L377

不过你们要切换sandbox，应该也不需要再花时间定位上面的问题