Repository: modelscope/evalscope
Issue: 测试用例报错
Root cause: Environment Incompatibility

测试代码

![Image](https://github.com/user-attachments/assets/54b0c8a9-3531-47f4-aa45-e82979ea6eaf)

报错

2025-03-20 15:04:31,802 - evalscope - INFO - Save the result to: ./outputs\20250320_150431\deepseek-r1_32b
2025-03-20 15:04:31,803 - evalscope - INFO - Starting benchmark...
2025-03-20 15:04:31,803 - evalscope - INFO - {
    "model": "deepseek-r1:32b",
    "model_id": "deepseek-r1:32b",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": null,
    "port": 8877,
    "url": "http://xxx:11334/v1/chat/completions",
    "headers": {},
    "connect_timeout": 600,
    "read_timeout": 600,
    "api_key": null,
    "number": 300,
    "parallel": 20,
    "rate": -1,
    "log_every_n_query": 10,
    "debug": false,
    "wandb_api_key": null,
    "name": null,
    "outputs_dir": "./outputs\\20250320_150431\\deepseek-r1_32b",
    "max_prompt_length": 131072,
    "min_prompt_length": 0,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "dataset": "openqa",
    "dataset_path": null,
    "frequency_penalty": null,
    "logprobs": null,
    "max_tokens": 2048,
    "min_tokens": null,
    "n_choices": null,
    "seed": 42,
    "stop": [],
    "stop_token_ids": [],
    "stream": true,
    "temperature": null,
    "top_p": null,
    "top_k": null
}
2025-03-20 15:04:41,452 - evalscope - INFO - Connection successful.
2025-03-20 15:04:41,459 - evalscope - ERROR - Exception in async function 'dispatch_requests_worker': [WinError 2] 系统找不到指定的文件。
Traceback (most recent call last):
  File "D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\LanguePath\Python\evalscope\evalscope\perf\benchmark.py", line 80, in dispatch_requests_worker
    total_queries = await dispatch_requests_from_dataset()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\LanguePath\Python\evalscope\evalscope\perf\benchmark.py", line 63, in dispatch_requests_from_dataset
    for messages in message_generator:
                    ^^^^^^^^^^^^^^^^^
  File "D:\LanguePath\Python\evalscope\evalscope\perf\plugin\datasets\openqa.py", line 21, in build_messages
    subprocess.call([
  File "C:\Python312\Lib\subprocess.py", line 389, in call
    with Popen(*popenargs, **kwargs) as p:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Python312\Lib\subprocess.py", line 1538, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] 系统找不到指定的文件。

Error
Traceback (most recent call last):
  File "D:LanguePath\Python\evalscope\evalscope\perf\utils\handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\LanguePath\Python\evalscope\evalscope\perf\benchmark.py", line 80, in dispatch_requests_worker
    total_queries = await dispatch_requests_from_dataset()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\LanguePath\Python\evalscope\evalscope\perf\benchmark.py", line 63, in dispatch_requests_from_dataset
    for messages in message_generator:
                    ^^^^^^^^^^^^^^^^^
  File "D:\LanguePath\Python\evalscope\evalscope\perf\plugin\datasets\openqa.py", line 21, in build_messages
    subprocess.call([
  File "C:\Python312\Lib\subprocess.py", line 389, in call
    with Popen(*popenargs, **kwargs) as p:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Python312\Lib\subprocess.py", line 1538, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] 系统找不到指定的文件。

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\LanguePath\Python\evalscope\tests\perf\test_perf.py", line 36, in test_run_perf_deepseek
    run_perf_benchmark(task_cfg)
  File "D:\LanguePath\Python\evalscope\evalscope\perf\main.py", line 39, in run_perf_benchmark
    loop.run_until_complete(benchmark(args))
  File "C:\Python312\Lib\asyncio\base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "C:\Python312\Lib\asyncio\base_events.py", line 641, in run_forever
    self._run_once()
  File "C:\Python312\Lib\asyncio\base_events.py", line 1986, in _run_once
    handle._run()
  File "C:\Python312\Lib\asyncio\events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py", line 20, in async_wrapper
    sys.exit(1)
SystemExit: 1



Ran 1 test in 9.669s

FAILED (errors=1)
2025-03-20 15:04:41,622 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-5' coro=<statistic_benchmark_metric_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,622 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-6' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,622 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-7' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,622 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-8' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,622 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-9' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,622 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-10' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,622 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-11' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,622 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-12' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,623 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-13' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,623 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-14' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,623 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-15' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,623 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-16' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,623 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-17' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,623 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-18' coro=<send_requests_worker() running at D:\\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,623 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-19' coro=<send_requests_worker() running at D:\\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,623 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-20' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,623 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-21' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,623 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-22' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,623 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-23' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,623 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-24' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,623 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-25' coro=<send_requests_worker() running at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14>>
2025-03-20 15:04:41,623 - asyncio - ERROR - Task exception was never retrieved
future: <Task finished name='Task-4' coro=<dispatch_requests_worker() done, defined at D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py:14> exception=SystemExit(1)>
Traceback (most recent call last):
  File "D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\LanguePath\Python\evalscope\evalscope\perf\benchmark.py", line 80, in dispatch_requests_worker
    total_queries = await dispatch_requests_from_dataset()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\LanguePath\Python\evalscope\evalscope\perf\benchmark.py", line 63, in dispatch_requests_from_dataset
    for messages in message_generator:
                    ^^^^^^^^^^^^^^^^^
  File "D:\LanguePath\Python\evalscope\evalscope\perf\plugin\datasets\openqa.py", line 21, in build_messages
    subprocess.call([
  File "C:\Python312\Lib\subprocess.py", line 389, in call
    with Popen(*popenargs, **kwargs) as p:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Python312\Lib\subprocess.py", line 1538, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] 系统找不到指定的文件。

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python312\Lib\unittest\case.py", line 58, in testPartExecutor
    yield
  File "C:\Python312\Lib\unittest\case.py", line 634, in run
    self._callTestMethod(testMethod)
  File "C:\Python312\Lib\unittest\case.py", line 589, in _callTestMethod
    if method() is not None:
       ^^^^^^^^
  File "D:\LanguePath\Python\evalscope\tests\perf\test_perf.py", line 36, in test_run_perf_deepseek
    run_perf_benchmark(task_cfg)
  File "D:\LanguePath\Python\evalscope\evalscope\perf\main.py", line 39, in run_perf_benchmark
    loop.run_until_complete(benchmark(args))
  File "C:\Python312\Lib\asyncio\base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "C:\Python312\Lib\asyncio\base_events.py", line 641, in run_forever
    self._run_once()
  File "C:\Python312\Lib\asyncio\base_events.py", line 1986, in _run_once
    handle._run()
  File "C:\Python312\Lib\asyncio\events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "D:\LanguePath\Python\evalscope\evalscope\perf\utils\handler.py", line 20, in async_wrapper
    sys.exit(1)
SystemExit: 1
sys:1: RuntimeWarning: coroutine 'statistic_benchmark_metric_worker' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
sys:1: RuntimeWarning: coroutine 'send_requests_worker' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

Process finished with exit code 1


dataset设置为longalpaca 试试呢，看报错是下载数据出问题了

> dataset设置为longalpaca 试试呢，看报错是下载数据出问题了

设置为longalpaca 可以了，openqa得手动下载数据集，指定路径才行