Repository: modelscope/evalscope
Issue: 使用 evalscope perf 压测模型遇到的问题
Root cause: Environment Incompatibility

## 问题描述 / Issue Description

使用 `--api local_vllm` 一直超时,这个需要我提前启动服务吗?

## 使用的工具 / Tools Used
- [x] Perf / 模型推理压测工具


## 执行的代码或指令 / Code or Commands Executed

```bash
root@devpod-1zry0b7qwzcl-1734254473990:/# CUDA_VISIBLE_DEVICES=0 evalscope perf  --rate 1  --model Qwen/Qwen2.5-7B-Instruct  --attn-implementation flash_attention_2  --log-every-n-query 5  --connect-timeout 6000  --read-timeout 6000  --max-tokens 2048  --min-tokens 2048  --api local_vllm  --dataset speed_benchmark  --debug --parallel 10
```

## 错误日志 / Error Log


```
2024-12-15 10:38:19,393 - datasets - INFO - PyTorch version 2.4.1+cu124 available.
2024-12-15 10:38:21,471 - evalscope - INFO - Save the result to: outputs/20241215_103821/Qwen2.5-7B-Instruct
2024-12-15 10:38:21,471 - evalscope - main.py - run_perf_benchmark - 31 - INFO - Starting benchmark...
2024-12-15 10:38:21,471 - evalscope - main.py - run_perf_benchmark - 32 - INFO - {
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "model_id": "Qwen2.5-7B-Instruct",
    "attn_implementation": "flash_attention_2",
    "api": "local_vllm",
    "tokenizer_path": null,
    "port": "8877",
    "url": "http://127.0.0.1:8877/v1/chat/completions",
    "headers": {
        "Authorization": "Bearer EMPTY"
    },
    "connect_timeout": 6000,
    "read_timeout": 6000,
    "api_key": "EMPTY",
    "number": null,
    "parallel": 10,
    "rate": 1,
    "log_every_n_query": 5,
    "debug": true,
    "wandb_api_key": null,
    "name": null,
    "outputs_dir": "outputs/20241215_103821/Qwen2.5-7B-Instruct",
    "max_prompt_length": 9223372036854775807,
    "min_prompt_length": 0,
    "prompt": null,
    "query_template": null,
    "dataset": "speed_benchmark",
    "dataset_path": null,
    "frequency_penalty": null,
    "logprobs": null,
    "max_tokens": 2048,
    "min_tokens": 2048,
    "n_choices": null,
    "seed": 42,
    "stop": null,
    "stop_token_ids": null,
    "stream": null,
    "temperature": null,
    "top_p": null
}
2024-12-15 10:38:21,473 - evalscope - http_client.py - on_request_start - 111 - DEBUG - Starting request: <TraceRequestStartParams(method='POST', url=URL('http://127.0.0.1:8877/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench', 'Authorization': 'Bearer EMPTY')>)>
2024-12-15 10:38:21,474 - evalscope - http_client.py - post - 106 - ERROR - Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
2024-12-15 10:38:21,474 - evalscope - http_client.py - test_connection - 159 - WARNING - Retrying...  <None> Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
/opt/conda/bin/python: Error while finding module specification for 'vllm.entrypoints.openai.api_server' (ModuleNotFoundError: No module named 'vllm')
2024-12-15 10:38:31,485 - evalscope - http_client.py - on_request_start - 111 - DEBUG - Starting request: <TraceRequestStartParams(method='POST', url=URL('http://127.0.0.1:8877/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench', 'Authorization': 'Bearer EMPTY')>)>
2024-12-15 10:38:31,486 - evalscope - http_client.py - post - 106 - ERROR - Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
2024-12-15 10:38:31,486 - evalscope - http_client.py - test_connection - 159 - WARNING - Retrying...  <None> Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
2024-12-15 10:38:41,498 - evalscope - http_client.py - on_request_start - 111 - DEBUG - Starting request: <TraceRequestStartParams(method='POST', url=URL('http://127.0.0.1:8877/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench', 'Authorization': 'Bearer EMPTY')>)>
2024-12-15 10:38:41,499 - evalscope - http_client.py - post - 106 - ERROR - Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
2024-12-15 10:38:41,499 - evalscope - http_client.py - test_connection - 159 - WARNING - Retrying...  <None> Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
2024-12-15 10:38:51,510 - evalscope - http_client.py - on_request_start - 111 - DEBUG - Starting request: <TraceRequestStartParams(method='POST', url=URL('http://127.0.0.1:8877/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench', 'Authorization': 'Bearer EMPTY')>)>
2024-12-15 10:38:51,512 - evalscope - http_client.py - post - 106 - ERROR - Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
2024-12-15 10:38:51,512 - evalscope - http_client.py - test_connection - 159 - WARNING - Retrying...  <None> Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
2024-12-15 10:39:01,523 - evalscope - http_client.py - on_request_start - 111 - DEBUG - Starting request: <TraceRequestStartParams(method='POST', url=URL('http://127.0.0.1:8877/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench', 'Authorization': 'Bearer EMPTY')>)>
2024-12-15 10:39:01,525 - evalscope - http_client.py - post - 106 - ERROR - Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
2024-12-15 10:39:01,525 - evalscope - http_client.py - test_connection - 159 - WARNING - Retrying...  <None> Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
2024-12-15 10:39:11,537 - evalscope - http_client.py - on_request_start - 111 - DEBUG - Starting request: <TraceRequestStartParams(method='POST', url=URL('http://127.0.0.1:8877/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench', 'Authorization': 'Bearer EMPTY')>)>
2024-12-15 10:39:11,538 - evalscope - http_client.py - post - 106 - ERROR - Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
2024-12-15 10:39:11,538 - evalscope - http_client.py - test_connection - 159 - WARNING - Retrying...  <None> Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]

```


## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
  - [x] Ubuntu

- Python版本 / Python Version:
  - [x] 3.11

## 其他信息 / Additional Information



日志说明环境里面没有vllm，请安装vllm后再使用

@Yunnglin 安装好 vllm 之后, 成功使用 Qwen2.5-7B 模型跑了一次 perf, 但是更换到 `Qwen/Qwen2.5-32B-Instruct-GGUF  ` 模型会报错

``` bash
CUDA_VISIBLE_DEVICES=0 evalscope perf  --rate 1  --model Qwen/Qwen2.5-32B-Instruct-GGUF  --attn-implementation flash_attention_2  --log-every-n-query 5  --connect-timeout 6000  --read-timeout 6000  --max-tokens 2048  --min-tokens 2048  --api local_vllm  --dataset speed_benchmark --parallel 10
```

``` txt
INFO 12-17 01:11:34 api_server.py:198] Started engine process with PID 4907
2024-12-17 01:11:40,072 - evalscope - ERROR - Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
2024-12-17 01:11:40,073 - evalscope - WARNING - Retrying...  <None> Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/vllm/vllm/entrypoints/openai/api_server.py", line 701, in <module>
    uvloop.run(run_server(args))
  File "/opt/conda/lib/python3.11/site-packages/uvloop/__init__.py", line 105, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/opt/conda/lib/python3.11/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/vllm/vllm/entrypoints/openai/api_server.py", line 667, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/opt/conda/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/vllm/vllm/entrypoints/openai/api_server.py", line 117, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/opt/conda/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/vllm/vllm/entrypoints/openai/api_server.py", line 209, in build_async_engine_client_from_engine_args
    engine_config = engine_args.create_engine_config()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vllm/vllm/engine/arg_utils.py", line 1029, in create_engine_config
    model_config = self.create_model_config()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vllm/vllm/engine/arg_utils.py", line 956, in create_model_config
    return ModelConfig(
           ^^^^^^^^^^^^
  File "/vllm/vllm/config.py", line 270, in __init__
    hf_config = get_config(self.model, trust_remote_code, revision,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vllm/vllm/transformers_utils/config.py", line 195, in get_config
    raise ValueError(f"No supported config format found in {model}")
ValueError: No supported config format found in Qwen/Qwen2.5-32B-Instruct-GGUF
ERROR 12-17 01:11:45 engine.py:366] No supported config format found in Qwen/Qwen2.5-32B-Instruct-GGUF
ERROR 12-17 01:11:45 engine.py:366] Traceback (most recent call last):
ERROR 12-17 01:11:45 engine.py:366]   File "/vllm/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
ERROR 12-17 01:11:45 engine.py:366]     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 12-17 01:11:45 engine.py:366]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-17 01:11:45 engine.py:366]   File "/vllm/vllm/engine/multiprocessing/engine.py", line 114, in from_engine_args
ERROR 12-17 01:11:45 engine.py:366]     engine_config = engine_args.create_engine_config(usage_context)
ERROR 12-17 01:11:45 engine.py:366]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-17 01:11:45 engine.py:366]   File "/vllm/vllm/engine/arg_utils.py", line 1029, in create_engine_config
ERROR 12-17 01:11:45 engine.py:366]     model_config = self.create_model_config()
ERROR 12-17 01:11:45 engine.py:366]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-17 01:11:45 engine.py:366]   File "/vllm/vllm/engine/arg_utils.py", line 956, in create_model_config
ERROR 12-17 01:11:45 engine.py:366]     return ModelConfig(
ERROR 12-17 01:11:45 engine.py:366]            ^^^^^^^^^^^^
ERROR 12-17 01:11:45 engine.py:366]   File "/vllm/vllm/config.py", line 270, in __init__
ERROR 12-17 01:11:45 engine.py:366]     hf_config = get_config(self.model, trust_remote_code, revision,
ERROR 12-17 01:11:45 engine.py:366]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-17 01:11:45 engine.py:366]   File "/vllm/vllm/transformers_utils/config.py", line 195, in get_config
ERROR 12-17 01:11:45 engine.py:366]     raise ValueError(f"No supported config format found in {model}")
ERROR 12-17 01:11:45 engine.py:366] ValueError: No supported config format found in Qwen/Qwen2.5-32B-Instruct-GGUF
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/vllm/vllm/engine/multiprocessing/engine.py", line 368, in run_mp_engine
    raise e
  File "/vllm/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vllm/vllm/engine/multiprocessing/engine.py", line 114, in from_engine_args
    engine_config = engine_args.create_engine_config(usage_context)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vllm/vllm/engine/arg_utils.py", line 1029, in create_engine_config
    model_config = self.create_model_config()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vllm/vllm/engine/arg_utils.py", line 956, in create_model_config
    return ModelConfig(
           ^^^^^^^^^^^^
  File "/vllm/vllm/config.py", line 270, in __init__
    hf_config = get_config(self.model, trust_remote_code, revision,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/vllm/vllm/transformers_utils/config.py", line 195, in get_config
    raise ValueError(f"No supported config format found in {model}")
ValueError: No supported config format found in Qwen/Qwen2.5-32B-Instruct-GGUF
2024-12-17 01:11:50,084 - evalscope - ERROR - Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
```

@Yunnglin 又遇到了另一个问题,麻烦您帮忙看一下

``` bash
CUDA_VISIBLE_DEVICES=0 evalscope perf  --rate 1  --model Qwen/Qwen-14B-Chat  --attn-implementation flash_attention_2  --log-every-n-query 5  --connect-timeout 6000  --read-timeout 6000  --max-tokens 2048  --min-tokens 2048  --api local_vllm  --dataset speed_benchmark --parallel 10
```

报错为

```txt
RuntimeError: Failed to load the model config. If the model is a custom model not yet available in the HuggingFace transformers library, consider setting `trust_remote_code=True` in LLM or using the `--trust-remote-code` flag in the CLI.
2024-12-17 01:17:58,709 - modelscope - WARNING - Model revision not specified, use revision: v1.0.8
Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen-14B-Chat
^CException ignored in atexit callback: <function _exit_function at 0x152007c81a80>
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/multiprocessing/util.py", line 360, in _exit_function
2024-12-17 01:18:01,047 - evalscope - INFO - Got signal SIGINT: exit
Traceback (most recent call last):
  File "/opt/conda/bin/evalscope", line 33, in <module>
    p.join()
    sys.exit(load_entry_point('evalscope', 'console_scripts', 'evalscope')())
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 149, in join
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/evalscope/evalscope/cli/cli.py", line 23, in run_cmd
    res = self._popen.wait(timeout)
    cmd.execute()
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/multiprocessing/popen_fork.py", line 43, in wait
  File "/opt/evalscope/evalscope/cli/start_perf.py", line 31, in execute
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
```

> @Yunnglin 安装好 vllm 之后, 成功使用 Qwen2.5-7B 模型跑了一次 perf, 但是更换到 `Qwen/Qwen2.5-32B-Instruct-GGUF ` 模型会报错
> 
> ```shell
> CUDA_VISIBLE_DEVICES=0 evalscope perf  --rate 1  --model Qwen/Qwen2.5-32B-Instruct-GGUF  --attn-implementation flash_attention_2  --log-every-n-query 5  --connect-timeout 6000  --read-timeout 6000  --max-tokens 2048  --min-tokens 2048  --api local_vllm  --dataset speed_benchmark --parallel 10
> ```
> 
> ```
> INFO 12-17 01:11:34 api_server.py:198] Started engine process with PID 4907
> 2024-12-17 01:11:40,072 - evalscope - ERROR - Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
> 2024-12-17 01:11:40,073 - evalscope - WARNING - Retrying...  <None> Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
> Traceback (most recent call last):
>   File "<frozen runpy>", line 198, in _run_module_as_main
>   File "<frozen runpy>", line 88, in _run_code
>   File "/vllm/vllm/entrypoints/openai/api_server.py", line 701, in <module>
>     uvloop.run(run_server(args))
>   File "/opt/conda/lib/python3.11/site-packages/uvloop/__init__.py", line 105, in run
>     return runner.run(wrapper())
>            ^^^^^^^^^^^^^^^^^^^^^
>   File "/opt/conda/lib/python3.11/asyncio/runners.py", line 118, in run
>     return self._loop.run_until_complete(task)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
>   File "/opt/conda/lib/python3.11/site-packages/uvloop/__init__.py", line 61, in wrapper
>     return await main
>            ^^^^^^^^^^
>   File "/vllm/vllm/entrypoints/openai/api_server.py", line 667, in run_server
>     async with build_async_engine_client(args) as engine_client:
>   File "/opt/conda/lib/python3.11/contextlib.py", line 210, in __aenter__
>     return await anext(self.gen)
>            ^^^^^^^^^^^^^^^^^^^^^
>   File "/vllm/vllm/entrypoints/openai/api_server.py", line 117, in build_async_engine_client
>     async with build_async_engine_client_from_engine_args(
>   File "/opt/conda/lib/python3.11/contextlib.py", line 210, in __aenter__
>     return await anext(self.gen)
>            ^^^^^^^^^^^^^^^^^^^^^
>   File "/vllm/vllm/entrypoints/openai/api_server.py", line 209, in build_async_engine_client_from_engine_args
>     engine_config = engine_args.create_engine_config()
>                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/vllm/vllm/engine/arg_utils.py", line 1029, in create_engine_config
>     model_config = self.create_model_config()
>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/vllm/vllm/engine/arg_utils.py", line 956, in create_model_config
>     return ModelConfig(
>            ^^^^^^^^^^^^
>   File "/vllm/vllm/config.py", line 270, in __init__
>     hf_config = get_config(self.model, trust_remote_code, revision,
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/vllm/vllm/transformers_utils/config.py", line 195, in get_config
>     raise ValueError(f"No supported config format found in {model}")
> ValueError: No supported config format found in Qwen/Qwen2.5-32B-Instruct-GGUF
> ERROR 12-17 01:11:45 engine.py:366] No supported config format found in Qwen/Qwen2.5-32B-Instruct-GGUF
> ERROR 12-17 01:11:45 engine.py:366] Traceback (most recent call last):
> ERROR 12-17 01:11:45 engine.py:366]   File "/vllm/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
> ERROR 12-17 01:11:45 engine.py:366]     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
> ERROR 12-17 01:11:45 engine.py:366]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> ERROR 12-17 01:11:45 engine.py:366]   File "/vllm/vllm/engine/multiprocessing/engine.py", line 114, in from_engine_args
> ERROR 12-17 01:11:45 engine.py:366]     engine_config = engine_args.create_engine_config(usage_context)
> ERROR 12-17 01:11:45 engine.py:366]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> ERROR 12-17 01:11:45 engine.py:366]   File "/vllm/vllm/engine/arg_utils.py", line 1029, in create_engine_config
> ERROR 12-17 01:11:45 engine.py:366]     model_config = self.create_model_config()
> ERROR 12-17 01:11:45 engine.py:366]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
> ERROR 12-17 01:11:45 engine.py:366]   File "/vllm/vllm/engine/arg_utils.py", line 956, in create_model_config
> ERROR 12-17 01:11:45 engine.py:366]     return ModelConfig(
> ERROR 12-17 01:11:45 engine.py:366]            ^^^^^^^^^^^^
> ERROR 12-17 01:11:45 engine.py:366]   File "/vllm/vllm/config.py", line 270, in __init__
> ERROR 12-17 01:11:45 engine.py:366]     hf_config = get_config(self.model, trust_remote_code, revision,
> ERROR 12-17 01:11:45 engine.py:366]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> ERROR 12-17 01:11:45 engine.py:366]   File "/vllm/vllm/transformers_utils/config.py", line 195, in get_config
> ERROR 12-17 01:11:45 engine.py:366]     raise ValueError(f"No supported config format found in {model}")
> ERROR 12-17 01:11:45 engine.py:366] ValueError: No supported config format found in Qwen/Qwen2.5-32B-Instruct-GGUF
> Process SpawnProcess-1:
> Traceback (most recent call last):
>   File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
>     self.run()
>   File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
>     self._target(*self._args, **self._kwargs)
>   File "/vllm/vllm/engine/multiprocessing/engine.py", line 368, in run_mp_engine
>     raise e
>   File "/vllm/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
>     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
>              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/vllm/vllm/engine/multiprocessing/engine.py", line 114, in from_engine_args
>     engine_config = engine_args.create_engine_config(usage_context)
>                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/vllm/vllm/engine/arg_utils.py", line 1029, in create_engine_config
>     model_config = self.create_model_config()
>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/vllm/vllm/engine/arg_utils.py", line 956, in create_model_config
>     return ModelConfig(
>            ^^^^^^^^^^^^
>   File "/vllm/vllm/config.py", line 270, in __init__
>     hf_config = get_config(self.model, trust_remote_code, revision,
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/vllm/vllm/transformers_utils/config.py", line 195, in get_config
>     raise ValueError(f"No supported config format found in {model}")
> ValueError: No supported config format found in Qwen/Qwen2.5-32B-Instruct-GGUF
> 2024-12-17 01:11:50,084 - evalscope - ERROR - Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
> ```

这个是因为vLL对GGUF模型支持还不完善，如果想使用GGUF模型可以用modelscope直接拉起服务[参考](https://modelscope.cn/docs/models/advanced-usage/ollama-integration)

> @Yunnglin 又遇到了另一个问题,麻烦您帮忙看一下
> 
> ```shell
> CUDA_VISIBLE_DEVICES=0 evalscope perf  --rate 1  --model Qwen/Qwen-14B-Chat  --attn-implementation flash_attention_2  --log-every-n-query 5  --connect-timeout 6000  --read-timeout 6000  --max-tokens 2048  --min-tokens 2048  --api local_vllm  --dataset speed_benchmark --parallel 10
> ```
> 
> 报错为
> 
> ```
> RuntimeError: Failed to load the model config. If the model is a custom model not yet available in the HuggingFace transformers library, consider setting `trust_remote_code=True` in LLM or using the `--trust-remote-code` flag in the CLI.
> 2024-12-17 01:17:58,709 - modelscope - WARNING - Model revision not specified, use revision: v1.0.8
> Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen-14B-Chat
> ^CException ignored in atexit callback: <function _exit_function at 0x152007c81a80>
> Traceback (most recent call last):
>   File "/opt/conda/lib/python3.11/multiprocessing/util.py", line 360, in _exit_function
> 2024-12-17 01:18:01,047 - evalscope - INFO - Got signal SIGINT: exit
> Traceback (most recent call last):
>   File "/opt/conda/bin/evalscope", line 33, in <module>
>     p.join()
>     sys.exit(load_entry_point('evalscope', 'console_scripts', 'evalscope')())
>   File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 149, in join
>              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/opt/evalscope/evalscope/cli/cli.py", line 23, in run_cmd
>     res = self._popen.wait(timeout)
>     cmd.execute()
>           ^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/opt/conda/lib/python3.11/multiprocessing/popen_fork.py", line 43, in wait
>   File "/opt/evalscope/evalscope/cli/start_perf.py", line 31, in execute
>     return self.poll(os.WNOHANG if timeout == 0.0 else 0)
> ```

这个是因为启动vllm时默认没有添加--trust-remote-code参数，现在可以拉main分支代码来试一下

> 这个是因为vLL对GGUF模型支持还不完善，如果想使用GGUF模型可以用modelscope直接拉起服务[参考](https://modelscope.cn/docs/models/advanced-usage/ollama-integration)

请问,使用 ollama 启动的模型,如何和 `evalscope perf` 集成到一起呢? 

我期望实现的是,使用 evalscope 构建一个用于基准测试的镜像, 自动化完成对模型的性能测试.

参考[使用API进行推理](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/speed_benchmark.html#api)，直接使用ollama启动模型服务，然后直接配置对应的url

@Yunnglin 

请问这个是版本兼容性问题吗? 最新版本的 `evalscope` 要配合什么版本的 `vllm` ? 请问有官方交流群吗?

```txt
2024-12-18 00:52:42,764 - evalscope - ERROR - Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
2024-12-18 00:52:42,764 - evalscope - WARNING - Retrying...  <None> Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/vllm/vllm/entrypoints/openai/api_server.py", line 30, in <module>
    from vllm.engine.multiprocessing.client import MQLLMEngineClient
  File "/vllm/vllm/engine/multiprocessing/__init__.py", line 7, in <module>
    from vllm import PoolingParams
ImportError: cannot import name 'PoolingParams' from 'vllm' (unknown location)

```



这边测试使用vllm 0.6.3没有问题

可以加下面企业微信
![image](https://github.com/user-attachments/assets/66799ccd-7c1c-49cf-b733-9e8c00675546)


@Yunnglin 很抱歉, 您提供的二维码显示群已满了 T.T

如何使用 `evalscope perf` 压测 70B 的大模型,使用单卡会 GPU OOM

如果不指定 `CUDA_VISIBLE_DEVICES` 应该会使用全部的 GPU 吧?

执行如下命令会报错:

``` bash
evalscope perf  --rate 1  --model LLM-Research/Llama-3.3-70B-Instruct  --attn-implementation flash_attention_2  --log-every-n-query 5  --connect-timeout 6000  --read-timeout 6000  --max-tokens 2048  --min-tokens 2048  --api local_vllm  --dataset speed_benchmark --parallel 10
```

``` txt
WARNING 12-19 01:19:22 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 96 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 12-19 01:19:22 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
(VllmWorkerProcess pid=27175) INFO 12-19 01:19:22 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
(VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
(VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]              ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/worker/worker.py", line 166, in init_device
(VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch.cuda.set_device(self.device)
(VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 420, in set_device
(VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch._C._cuda_setDevice(device)
(VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 300, in _lazy_init
(VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     raise RuntimeError(
(VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
(VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] 
(VllmWorkerProcess pid=27176) INFO 12-19 01:19:22 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=27174) INFO 12-19 01:19:22 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
(VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
(VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]              ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
(VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/worker/worker.py", line 166, in init_device
(VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch.cuda.set_device(self.device)
(VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
(VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 420, in set_device
(VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch._C._cuda_setDevice(device)
(VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]              ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 300, in _lazy_init
(VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/worker/worker.py", line 166, in init_device
(VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     raise RuntimeError(
(VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch.cuda.set_device(self.device)
(VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
(VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 420, in set_device
(VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] 
(VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch._C._cuda_setDevice(device)
(VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 300, in _lazy_init
(VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     raise RuntimeError(
(VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
(VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] 
2024-12-19 01:19:29,447 - evalscope - ERROR - Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
```

> @Yunnglin 很抱歉, 您提供的二维码显示群已满了 T.T

你可以扫描readme里面的二维码，已经更新了

> 如何使用 `evalscope perf` 压测 70B 的大模型,使用单卡会 GPU OOM
> 
> 如果不指定 `CUDA_VISIBLE_DEVICES` 应该会使用全部的 GPU 吧?
> 
> 执行如下命令会报错:
> 
> ```shell
> evalscope perf  --rate 1  --model LLM-Research/Llama-3.3-70B-Instruct  --attn-implementation flash_attention_2  --log-every-n-query 5  --connect-timeout 6000  --read-timeout 6000  --max-tokens 2048  --min-tokens 2048  --api local_vllm  --dataset speed_benchmark --parallel 10
> ```
> 
> ```
> WARNING 12-19 01:19:22 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 96 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
> INFO 12-19 01:19:22 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
> (VllmWorkerProcess pid=27175) INFO 12-19 01:19:22 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
> (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
> (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
> (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
> (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]              ^^^^^^^^^^^^^^^^^^^^^^^^^
> (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/worker/worker.py", line 166, in init_device
> (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch.cuda.set_device(self.device)
> (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 420, in set_device
> (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch._C._cuda_setDevice(device)
> (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 300, in _lazy_init
> (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     raise RuntimeError(
> (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
> (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] 
> (VllmWorkerProcess pid=27176) INFO 12-19 01:19:22 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
> (VllmWorkerProcess pid=27174) INFO 12-19 01:19:22 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
> (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
> (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
> (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
> (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]              ^^^^^^^^^^^^^^^^^^^^^^^^^
> (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
> (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/worker/worker.py", line 166, in init_device
> (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch.cuda.set_device(self.device)
> (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
> (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 420, in set_device
> (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
> (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch._C._cuda_setDevice(device)
> (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]              ^^^^^^^^^^^^^^^^^^^^^^^^^
> (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 300, in _lazy_init
> (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/worker/worker.py", line 166, in init_device
> (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     raise RuntimeError(
> (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch.cuda.set_device(self.device)
> (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
> (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 420, in set_device
> (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] 
> (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch._C._cuda_setDevice(device)
> (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 300, in _lazy_init
> (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     raise RuntimeError(
> (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
> (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] 
> 2024-12-19 01:19:29,447 - evalscope - ERROR - Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
> ```

这块我们再看看修改一下，暂时可以先使用vllm自行启动模型推理服务，指定url来进行测试

> > 如何使用 `evalscope perf` 压测 70B 的大模型,使用单卡会 GPU OOM
> > 如果不指定 `CUDA_VISIBLE_DEVICES` 应该会使用全部的 GPU 吧?
> > 执行如下命令会报错:
> > ```shell
> > evalscope perf  --rate 1  --model LLM-Research/Llama-3.3-70B-Instruct  --attn-implementation flash_attention_2  --log-every-n-query 5  --connect-timeout 6000  --read-timeout 6000  --max-tokens 2048  --min-tokens 2048  --api local_vllm  --dataset speed_benchmark --parallel 10
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > ```
> > WARNING 12-19 01:19:22 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 96 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
> > INFO 12-19 01:19:22 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
> > (VllmWorkerProcess pid=27175) INFO 12-19 01:19:22 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
> > (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
> > (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
> > (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
> > (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]              ^^^^^^^^^^^^^^^^^^^^^^^^^
> > (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/worker/worker.py", line 166, in init_device
> > (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch.cuda.set_device(self.device)
> > (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 420, in set_device
> > (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch._C._cuda_setDevice(device)
> > (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 300, in _lazy_init
> > (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     raise RuntimeError(
> > (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
> > (VllmWorkerProcess pid=27175) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] 
> > (VllmWorkerProcess pid=27176) INFO 12-19 01:19:22 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
> > (VllmWorkerProcess pid=27174) INFO 12-19 01:19:22 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
> > (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
> > (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
> > (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
> > (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]              ^^^^^^^^^^^^^^^^^^^^^^^^^
> > (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
> > (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/worker/worker.py", line 166, in init_device
> > (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch.cuda.set_device(self.device)
> > (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
> > (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 420, in set_device
> > (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
> > (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch._C._cuda_setDevice(device)
> > (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]              ^^^^^^^^^^^^^^^^^^^^^^^^^
> > (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 300, in _lazy_init
> > (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/vllm/vllm/worker/worker.py", line 166, in init_device
> > (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     raise RuntimeError(
> > (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch.cuda.set_device(self.device)
> > (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
> > (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 420, in set_device
> > (VllmWorkerProcess pid=27176) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] 
> > (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     torch._C._cuda_setDevice(device)
> > (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]   File "/opt/conda/lib/python3.11/site-packages/torch/cuda/__init__.py", line 300, in _lazy_init
> > (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231]     raise RuntimeError(
> > (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
> > (VllmWorkerProcess pid=27174) ERROR 12-19 01:19:22 multiproc_worker_utils.py:231] 
> > 2024-12-19 01:19:29,447 - evalscope - ERROR - Cannot connect to host 127.0.0.1:8877 ssl:default [Connect call failed ('127.0.0.1', 8877)]
> > ```
> 
> 这块我们再看看修改一下，暂时可以先使用vllm自行启动模型推理服务，指定url来进行测试

可能与这个有关吗 [Distributed Inference and Serving](https://docs.vllm.ai/en/latest/serving/distributed_serving.html#details-for-distributed-inference-and-serving)

@Yunnglin 

在使用 H100 x4 压测 Llama-3.3-70B 

### 启动服务

``` bash
vllm serve \
    /root/.cache/modelscope/hub/LLM-Research/Llama-3.3-70B-Instruct \
    --tensor-parallel-size 4
```

### 启动压测

``` bash
evalscope perf \
    --rate 10 \
    --url http://127.0.0.1:8000/v1/completions \
    --model /root/.cache/modelscope/hub/LLM-Research/Llama-3.3-70B-Instruct \
    --log-every-n-query 5 \
    --connect-timeout 6000 \
    --read-timeout 6000 \
    --max-tokens 2048 \
    --min-tokens 2048 \
    --api openai \
    --dataset speed_benchmark \
    --debug \
    --parallel 4
```

### Benchmarking summary:

| Key                               | Value                                                            |
| --------------------------------- | ---------------------------------------------------------------- |
| Time taken for tests (s)          | 121.531                                                          |
| Number of concurrency             | 4                                                                |
| Total requests                    | 8                                                                |
| Succeed requests                  | 8                                                                |
| Failed requests                   | 0                                                                |
| Throughput(average tokens/s)      | 134.813                                                          |
| Average QPS                       | 0.066                                                            |
| Average latency (s)               | 58.891                                                           |
| Average time to first token (s)   | 58.891                                                           |
| Average time per output token (s) | 0.00742                                                          |
| Average input tokens per request  | 25601.5                                                          |
| Average output tokens per request | 2048.0                                                           |
| Average package latency (s)       | 58.891                                                           |
| Average package per request       | 1.0                                                              |
| Expected number of requests       | 8                                                                |
| Result DB path                    | outputs/20241219_054843/Llama-3.3-70B-Instruct/benchmark_data.db |

### Speed Benchmark Results:

| Prompt Tokens | Speed(tokens/s) |
| ------------- | --------------- |
| 3             | 48.79           |
| 12289         | 45.75           |
| 28673         | 27.79           |
| 61441         | 27.27           |

### 问题

1. 如何理解表格一和表格二的结果
* Throughput(average tokens/s) :  134.813

2. 首 Token 时间 ` Average time to first token (s)  :  58.891` 与直觉相违背

使用 curl 测试的结果如下

``` bash
root@devpod-1zuahcf0adx0-1734396516660:~# time curl --location --request POST 'http://localhost:8000/v1/chat/completions' --header 'User-Agent: Apifox/1.0.0 (https://apifox.com)' --header 'Content-Type: application/json' --data-raw '{
    "messages": [

        {
            "content": "Write a quick sort in c programming language",
            "role": "user"
        }
    ],
    "model": "/root/.cache/modelscope/hub/LLM-Research/Llama-3.3-70B-Instruct"
}'
{"id":"chat-e1f9e28668314034ad483372874b2182","object":"chat.completion","created":1734588926,"model":"/root/.cache/modelscope/hub/LLM-Research/Llama-3.3-70B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Here's an implementation of the quicksort algorithm in C:\n\n```c\n#include <stdio.h>\n\n// Function to swap two elements\nvoid swap(int *a, int *b) {\n    int temp = *a;\n    *a = *b;\n    *b = temp;\n}\n\n// Function to partition the array and return the pivot index\nint partition(int arr[], int low, int high) {\n    int pivot = arr[high]; // Choose the last element as the pivot\n    int i = low - 1;\n\n    for (int j = low; j < high; j++) {\n        if (arr[j] < pivot) {\n            i++;\n            swap(&arr[i], &arr[j]);\n        }\n    }\n\n    swap(&arr[i + 1], &arr[high]);\n    return i + 1;\n}\n\n// Function to perform quicksort\nvoid quicksort(int arr[], int low, int high) {\n    if (low < high) {\n        int pivot = partition(arr, low, high);\n\n        quicksort(arr, low, pivot - 1);\n        quicksort(arr, pivot + 1, high);\n    }\n}\n\n// Function to print the array\nvoid printArray(int arr[], int size) {\n    for (int i = 0; i < size; i++) {\n        printf(\"%d \", arr[i]);\n    }\n    printf(\"\\n\");\n}\n\n// Main function\nint main() {\n    int arr[] = {5, 2, 9, 1, 7, 3, 6, 8, 4};\n    int n = sizeof(arr) / sizeof(arr[0]);\n\n    printf(\"Original array: \");\n    printArray(arr, n);\n\n    quicksort(arr, 0, n - 1);\n\n    printf(\"Sorted array: \");\n    printArray(arr, n);\n\n    return 0;\n}\n```\n\nThis implementation uses the \"Lomuto\" partition scheme, which is a variation of the standard \"Hoare\" partition scheme that is slightly faster and more efficient in practice.\n\nHere's a brief explanation of how the code works:\n\n1.  The `swap` function swaps two elements in the array.\n2.  The `partition` function rearranges the array such that all elements less than the pivot are on the left of the pivot, and all elements greater than the pivot are on the right. The function returns the index of the pivot element.\n3.  The `quicksort` function recursively sorts the subarrays on either side of the pivot element.\n4.  The `printArray` function prints the elements of the array.\n5.  In the `main` function, we create an example array, print it, sort it using quicksort, and then print the sorted array.\n\nWhen you run this code, it will output the original array and the sorted array:\n\n```\nOriginal array: 5 2 9 1 7 3 6 8 4\nSorted array: 1 2 3 4 5 6 7 8 9\n```","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":43,"total_tokens":680,"completion_tokens":637},"prompt_logprobs":null}
real    0m11.841s
user    0m0.004s
sys     0m0.005s
```



> > @Yunnglin 很抱歉, 您提供的二维码显示群已满了 T.T
> 
> 你可以扫描readme里面的二维码，已经更新了

同样提示群已满 

请使用rate为1，parallel为1 进行speed_benchmark

- 表格1计算所有请求的Throughput；表格2计算的是单个请求内的Throughput。模型服务可能会同时并发处理请求，导致总的Throughput比较高。
- parallel设置为4会导致有些请求排队，首token时间变长


> 请使用rate为1，parallel为1 进行speed_benchmark
> 
> parallel设置为4会导致有些请求排队，首token时间变长

我也测试了 rate=1 , parallel=1 的情况

<!--StartFragment--></meta><byte-sheet-html-origin data-id="1734592804675" data-version="4" data-is-embed="true" data-grid-line-hidden="false">
Average time to first token (s) | 43.999
-- | --


</byte-sheet-html-origin><!--EndFragment-->

相比 `58.891` 确实有一定的提升.

请问在压测的过程中, 有哪些参数值得调整吗? 

主要能调的参数是[请求控制](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/parameters.html#id4)，可以参考一下

readme里二维码已更新

> 由于长时间未收到活动，我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！