Repository: modelscope/evalscope
Issue: 单并发多请求的压测指标偏低
Root cause: Algorithmic Error

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [ &#10004;] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [ &#10004;] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [ &#10004;] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述

我在压测由vllm启动的Qwen2.5-32B-Instruct-GPTQ-Int8模型，观察到在 --parallel 1 + --number 1 手动请求5次取得的平均值 和 --parallel 1 + --number 5 一次性请求5次 时结果指标（TTFT(s) 和 Output (tok/s)）以及输入指标（输入token数量）差异很大.
- 【 --parallel 1 + --number 1】：输入参数中可以设置的最大token数量和输出token数量限制分别为：30000和500；输出指标中，TTFT约为 0.25，Output token throughput (tok/s) 约为 20.
- 【--parallel 1 + --number 5】：输入参数中可以设置的最大token数量和输出token数量限制分别为：28000和500；输出指标中，TTFT约为 16，Output token throughput (tok/s) 约为 13.

## EvalScope 版本（必填）
v0.16.1

## 使用的工具
- [ &#10004;] Perf / 模型推理压测工具

## 执行的代码或指令
### 【--parallel 1 + --number 1】对应指令
```
evalscope perf --model Qwen/Qwen2.5-32B-Instruct-GPTQ-Int8 --parallel 1 --number 1 --url http://192.168.0.55:8000/v1/chat/completions --api openai --api-key EMPTY --dataset random --min-prompt-length 30000 --max-prompt-length 30000 --min-tokens 500 --max-tokens 500 --tokenizer-path models/Qwen2.5-32B-Instruct-GPTQ-Int8 --swanlab-api-key local
```
### 【--parallel 1 + --number 5】对应指令
```
evalscope perf --model Qwen/Qwen2.5-32B-Instruct-GPTQ-Int8 --parallel 1 --number 5 --url http://192.168.0.55:8000/v1/chat/completions --api openai --api-key EMPTY --dataset random --min-prompt-length 28000 --max-prompt-length 28000 --min-tokens 500 --max-tokens 500 --tokenizer-path models/Qwen2.5-32B-Instruct-GPTQ-Int8 --swanlab-api-key local
```

## 错误日志
### --parallel 1 + --number 1】输出日志
```
    "attn_implementation": null
    "api": "openai"
    "tokenizer_path": "models/Qwen2.5-32B-Instruct-GPTQ-Int8"
    "port": 8877
    "url": "http://192.168.0.55:8000/v1/chat/completions"
    "headers": 
        "Authorization": "Bearer EMPTY
    }
    "connect_timeout": 600
    "read_timeout": 600
    "api_key": "EMPTY"
    "no_test_connection": false
    "number": 1
    "parallel": 1
    "rate": -1
    "log_every_n_query": 10
    "debug": false
    "wandb_api_key": null
    "swanlab_api_key": "local"
    "name": null
    "outputs_dir": "outputs/20250617_111954/Qwen2.5-32B-Instruct-GPTQ-Int8"
    "max_prompt_length": 30000
    "min_prompt_length": 30000
    "prefix_length": 0
    "prompt": null
    "query_template": null
    "apply_chat_template": true
    "dataset": "random"
    "dataset_path": null
    "frequency_penalty": null
    "logprobs": null
    "max_tokens": 500
    "min_tokens": 500
    "n_choices": null
    "seed": 0
    "stop": null
    "stop_token_ids": null
    "stream": true
    "temperature": 0.0
    "top_p": null
    "top_k": null
    "extra_args": {
2025-06-17 11:19:59,473 - evalscope - INFO - Test connection successful
2025-06-17 11:20:00,752 - evalscope - INFO - Save the data base to: outputs/20250617_111954/Qwen2.5-32B-Instruct-GPTQ-Int8/benchmark_data.d
Processing: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:24<00:00, 24.35s/it
2025-06-17 11:20:25,139 - evalscope - INFO -
Benchmarking summary
+-----------------------------------+------------
| Key                               |      Value 
+===================================+============
| Time taken for tests (s)          |    24.0511 
+-----------------------------------+------------
| Number of concurrency             |     1      
+-----------------------------------+------------
| Total requests                    |     1      
+-----------------------------------+------------
| Succeed requests                  |     1      
+-----------------------------------+------------
| Failed requests                   |     0      
+-----------------------------------+------------
| Output token throughput (tok/s)   |    20.789  
+-----------------------------------+------------
| Total token throughput (tok/s)    |  1334.62   
+-----------------------------------+------------
| Request throughput (req/s)        |     0.0416 
+-----------------------------------+------------
| Average latency (s)               |    24.0502 
+-----------------------------------+------------
| Average time to first token (s)   |     0.253  
+-----------------------------------+------------
| Average time per output token (s) |     0.0476 
+-----------------------------------+------------
| Average input tokens per request  | 31599      
+-----------------------------------+------------
| Average output tokens per request |   500      
+-----------------------------------+------------
| Average package latency (s)       |     0.0476 
+-----------------------------------+------------
| Average package per request       |   500      
+-----------------------------------+------------
2025-06-17 11:20:25,141 - evalscope - INFO -
Percentile results
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------
| Percentiles | TTFT (s) | ITL (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Output (tok/s) | Total (tok/s) 
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------
|     10%     |  0.253   | 0.0475  |  0.0476  |   24.0502   |    31599     |      500      |    20.7898     |   1334.6656   
|     25%     |  0.253   | 0.0476  |  0.0476  |   24.0502   |    31599     |      500      |    20.7898     |   1334.6656   
|     50%     |  0.253   | 0.0477  |  0.0476  |   24.0502   |    31599     |      500      |    20.7898     |   1334.6656   
|     66%     |  0.253   | 0.0477  |  0.0476  |   24.0502   |    31599     |      500      |    20.7898     |   1334.6656   
|     75%     |  0.253   | 0.0478  |  0.0476  |   24.0502   |    31599     |      500      |    20.7898     |   1334.6656   
|     80%     |  0.253   | 0.0478  |  0.0476  |   24.0502   |    31599     |      500      |    20.7898     |   1334.6656   
|     90%     |  0.253   | 0.0479  |  0.0476  |   24.0502   |    31599     |      500      |    20.7898     |   1334.6656   
|     95%     |  0.253   |  0.048  |  0.0476  |   24.0502   |    31599     |      500      |    20.7898     |   1334.6656   
|     98%     |  0.253   | 0.0481  |  0.0476  |   24.0502   |    31599     |      500      |    20.7898     |   1334.6656   
|     99%     |  0.253   | 0.0481  |  0.0476  |   24.0502   |    31599     |      500      |    20.7898     |   1334.6656   
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------
2025-06-17 11:20:25,141 - evalscope - INFO - Save the summary to: outputs/20250617_111954/Qwen2.5-32B-Instruct-GPTQ-Int
swanlab: Experiment Qwen2.5-32B-Instruct-GPTQ-Int8_20250617_111954 has complete
swanlab: 🌟 Run `swanlab watch /home/zmy/llm_test/outputs` to view SwanLab Experiment Dashboard locall
```
### 【--parallel 1 + --number 5】输出日志
```
    "attn_implementation": null
    "api": "openai"
    "tokenizer_path": "models/Qwen2.5-32B-Instruct-GPTQ-Int8"
    "port": 8877
    "url": "http://192.168.0.55:8000/v1/chat/completions"
    "headers": 
        "Authorization": "Bearer EMPTY
    }
    "connect_timeout": 600
    "read_timeout": 600
    "api_key": "EMPTY"
    "no_test_connection": false
    "number": 5
    "parallel": 1
    "rate": -1
    "log_every_n_query": 10
    "debug": false
    "wandb_api_key": null
    "swanlab_api_key": "local"
    "name": null
    "outputs_dir": "outputs/20250617_110545/Qwen2.5-32B-Instruct-GPTQ-Int8"
    "max_prompt_length": 28000
    "min_prompt_length": 28000
    "prefix_length": 0
    "prompt": null
    "query_template": null
    "apply_chat_template": true
    "dataset": "random"
    "dataset_path": null
    "frequency_penalty": null
    "logprobs": null
    "max_tokens": 500
    "min_tokens": 500
    "n_choices": null
    "seed": 0
    "stop": null
    "stop_token_ids": null
    "stream": true
    "temperature": 0.0
    "top_p": null
    "top_k": null
    "extra_args": {
2025-06-17 11:05:50,963 - evalscope - INFO - Test connection successful
2025-06-17 11:05:52,244 - evalscope - INFO - Save the data base to: outputs/20250617_110545/Qwen2.5-32B-Instruct-GPTQ-Int8/benchmark_data.d
Processing: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [03:19<00:00, 39.93s/it
2025-06-17 11:09:12,017 - evalscope - INFO -
Benchmarking summary
+-----------------------------------+------------
| Key                               |      Value 
+===================================+============
| Time taken for tests (s)          |   199.623  
+-----------------------------------+------------
| Number of concurrency             |     1      
+-----------------------------------+------------
| Total requests                    |     5      
+-----------------------------------+------------
| Succeed requests                  |     5      
+-----------------------------------+------------
| Failed requests                   |     0      
+-----------------------------------+------------
| Output token throughput (tok/s)   |    12.5236 
+-----------------------------------+------------
| Total token throughput (tok/s)    |   750.631  
+-----------------------------------+------------
| Request throughput (req/s)        |     0.025  
+-----------------------------------+------------
| Average latency (s)               |    39.8724 
+-----------------------------------+------------
| Average time to first token (s)   |    16.3791 
+-----------------------------------+------------
| Average time per output token (s) |     0.047  
+-----------------------------------+------------
| Average input tokens per request  | 29468.6    
+-----------------------------------+------------
| Average output tokens per request |   500      
+-----------------------------------+------------
| Average package latency (s)       |     0.047  
+-----------------------------------+------------
| Average package per request       |   500      
+-----------------------------------+------------
2025-06-17 11:09:12,019 - evalscope - INFO -
Percentile results
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------
| Percentiles | TTFT (s) | ITL (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Output (tok/s) | Total (tok/s) 
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------
|     10%     | 15.4764  | 0.0466  |  0.0466  |   38.7973   |    28152     |      500      |    11.9067     |   737.9796    
|     25%     | 15.5149  | 0.0467  |  0.0466  |   38.8249   |    28152     |      500      |    12.5221     |   738.5051    
|     50%     | 16.3322  |  0.047  |  0.047   |   39.8174   |    29496     |      500      |    12.5573     |   751.3266    
|     66%     | 16.4477  | 0.0471  |  0.047   |   39.9294   |    29500     |      500      |    12.8783     |   753.3396    
|     75%     | 16.4477  | 0.0472  |  0.047   |   39.9294   |    29500     |      500      |    12.8783     |   753.3396    
|     80%     | 18.1242  | 0.0475  |  0.0477  |   41.9931   |    32043     |      500      |    12.8875     |   774.9607    
|     90%     | 18.1242  | 0.0478  |  0.0477  |   41.9931   |    32043     |      500      |    12.8875     |   774.9607    
|     95%     | 18.1242  | 0.0479  |  0.0477  |   41.9931   |    32043     |      500      |    12.8875     |   774.9607    
|     98%     | 18.1242  |  0.048  |  0.0477  |   41.9931   |    32043     |      500      |    12.8875     |   774.9607    
|     99%     | 18.1242  | 0.0481  |  0.0477  |   41.9931   |    32043     |      500      |    12.8875     |   774.9607    
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------
2025-06-17 11:09:12,020 - evalscope - INFO - Save the summary to: outputs/20250617_110545/Qwen2.5-32B-Instruct-GPTQ-Int
swanlab: Experiment Qwen2.5-32B-Instruct-GPTQ-Int8_20250617_110545 has complete
swanlab: 🌟 Run `swanlab watch /home/zmy/llm_test/outputs` to view SwanLab Experiment Dashboard locall

```
## 运行环境

- 操作系统：ubuntu20.04
- Python版本：3.10.16

## 其他信息

如果有其他相关信息，请在此处提供。


“--parallel 1 + --number 1 手动请求5次”是连续的请求吗，中间有间隔吗

> “--parallel 1 + --number 1 手动请求5次”是连续的请求吗，中间有间隔吗

有，就是当我看到上次的请求已经生成了结果后，我再输入指令发起新的请求，间隔时间不固定。

你的这些请求prompt太长了，可能把vllm的kv cache打满了；尝试降低prompt长度再试一下呢

> 你的这些请求prompt太长了，可能把vllm的kv cache打满了；尝试降低prompt长度再试一下呢

首先，我尝试了短prompt 对应的两种用例执行，结果显示两者得到的性能指标差异不大，属正常范围。
其次，我通过脚本编写批量的“--parallel 1 + --number 1” 对应长prompt请求，得到的多次结果差异并不是很大。
相请教一下，使用参数“--parallel 1 + --number 5”执行的过程与我编写的批量执行脚本是存在什么实质性差异吗？批量执行脚本的输入prompt也是级长的呢。
执行指令：bash test.sh, 其中批量测试脚本 test.sh内容如下：
```
bash single/vllm/test_vllm_Qwen2.5-32B-Instruct-GPTQ-Int8.sh
bash single/vllm/test_vllm_Qwen2.5-32B-Instruct-GPTQ-Int8.sh
bash single/vllm/test_vllm_Qwen2.5-32B-Instruct-GPTQ-Int8.sh
bash single/vllm/test_vllm_Qwen2.5-32B-Instruct-GPTQ-Int8.sh
bash single/vllm/test_vllm_Qwen2.5-32B-Instruct-GPTQ-Int8.sh
```
其中文件 single/vllm/test_vllm_Qwen2.5-32B-Instruct-GPTQ-Int8.sh 对应内容：
```
model_id=Qwen/Qwen2.5-32B-Instruct-GPTQ-Int8
input_length=30000
output_length=500

evalscope perf \
  --model ${model_id} \
  --parallel 1 \
  --number 1 \
  --url http://192.168.0.56:8000/v1/chat/completions --api openai \
  --api-key EMPTY \
  --dataset random \
  --min-prompt-length ${input_length} --max-prompt-length ${input_length} \
  --min-tokens ${output_length} --max-tokens ${output_length} \
  --tokenizer-path models/Qwen2.5-32B-Instruct-GPTQ-Int8 \
  --swanlab-api-key local
```
测试结果日志：
```
2025-06-18 16:08:51,934 - evalscope - INFO - Test connection successful.
2025-06-18 16:08:53,145 - evalscope - INFO - Save the data base to: outputs/20250618_160846/Qwen2.5-32B-Instruct-GPTQ-Int8/benchmark_data.db
Processing: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:24<00:00, 24.29s/it]
2025-06-18 16:09:17,445 - evalscope - INFO - 
Benchmarking summary:
+-----------------------------------+------------+
| Key                               |      Value |
+===================================+============+
| Time taken for tests (s)          |    24.0719 |
+-----------------------------------+------------+
| Number of concurrency             |     1      |
+-----------------------------------+------------+
| Total requests                    |     1      |
+-----------------------------------+------------+
| Succeed requests                  |     1      |
+-----------------------------------+------------+
| Failed requests                   |     0      |
+-----------------------------------+------------+
| Output token throughput (tok/s)   |    20.7711 |
+-----------------------------------+------------+
| Total token throughput (tok/s)    |  1333.46   |
+-----------------------------------+------------+
| Request throughput (req/s)        |     0.0415 |
+-----------------------------------+------------+
| Average latency (s)               |    24.0713 |
+-----------------------------------+------------+
| Average time to first token (s)   |     0.2846 |
+-----------------------------------+------------+
| Average time per output token (s) |     0.0476 |
+-----------------------------------+------------+
| Average input tokens per request  | 31599      |
+-----------------------------------+------------+
| Average output tokens per request |   500      |
+-----------------------------------+------------+
| Average package latency (s)       |     0.0476 |
+-----------------------------------+------------+
| Average package per request       |   500      |
+-----------------------------------+------------+
2025-06-18 16:09:17,447 - evalscope - INFO - 
Percentile results:
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| Percentiles | TTFT (s) | ITL (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Output (tok/s) | Total (tok/s) |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
|     10%     |  0.2846  | 0.0475  |  0.0476  |   24.0713   |    31599     |      500      |    20.7716     |   1333.4979   |
|     25%     |  0.2846  | 0.0476  |  0.0476  |   24.0713   |    31599     |      500      |    20.7716     |   1333.4979   |
|     50%     |  0.2846  | 0.0477  |  0.0476  |   24.0713   |    31599     |      500      |    20.7716     |   1333.4979   |
|     66%     |  0.2846  | 0.0477  |  0.0476  |   24.0713   |    31599     |      500      |    20.7716     |   1333.4979   |
|     75%     |  0.2846  | 0.0477  |  0.0476  |   24.0713   |    31599     |      500      |    20.7716     |   1333.4979   |
|     80%     |  0.2846  | 0.0477  |  0.0476  |   24.0713   |    31599     |      500      |    20.7716     |   1333.4979   |
|     90%     |  0.2846  | 0.0478  |  0.0476  |   24.0713   |    31599     |      500      |    20.7716     |   1333.4979   |
|     95%     |  0.2846  | 0.0479  |  0.0476  |   24.0713   |    31599     |      500      |    20.7716     |   1333.4979   |
|     98%     |  0.2846  |  0.048  |  0.0476  |   24.0713   |    31599     |      500      |    20.7716     |   1333.4979   |
|     99%     |  0.2846  |  0.048  |  0.0476  |   24.0713   |    31599     |      500      |    20.7716     |   1333.4979   |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
2025-06-18 16:09:17,447 - evalscope - INFO - Save the summary to: outputs/20250618_160846/Qwen2.5-32B-Instruct-GPTQ-Int8
swanlab: Experiment Qwen2.5-32B-Instruct-GPTQ-Int8_20250618_160846 has completed
swanlab: 🌟 Run `swanlab watch /home/zmy/llm_test/outputs` to view SwanLab Experiment Dashboard locally
2025-06-18 16:09:21,500 - evalscope - INFO - Save the result to: outputs/20250618_160921/Qwen2.5-32B-Instruct-GPTQ-Int8
swanlab: Tracking run with swanlab version 0.6.1
swanlab: Run data will be saved locally in /home/zmy/llm_test/outputs/run-20250618_160926-7a024204
swanlab: 🌟 Run `swanlab watch /home/zmy/llm_test/outputs` to view SwanLab Experiment Dashboard locally
2025-06-18 16:09:27,117 - evalscope - INFO - Starting benchmark with args: 
2025-06-18 16:09:27,117 - evalscope - INFO - {
    "model": "Qwen/Qwen2.5-32B-Instruct-GPTQ-Int8",
    "model_id": "Qwen2.5-32B-Instruct-GPTQ-Int8",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": "models/Qwen2.5-32B-Instruct-GPTQ-Int8",
    "port": 8877,
    "url": "http://192.168.0.56:8000/v1/chat/completions",
    "headers": {
        "Authorization": "Bearer EMPTY"
    },
    "connect_timeout": 600,
    "read_timeout": 600,
    "api_key": "EMPTY",
    "no_test_connection": false,
    "number": 1,
    "parallel": 1,
    "rate": -1,
    "log_every_n_query": 10,
    "debug": false,
    "wandb_api_key": null,
    "swanlab_api_key": "local",
    "name": null,
    "outputs_dir": "outputs/20250618_160921/Qwen2.5-32B-Instruct-GPTQ-Int8",
    "max_prompt_length": 30000,
    "min_prompt_length": 30000,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": true,
    "dataset": "random",
    "dataset_path": null,
    "frequency_penalty": null,
    "logprobs": null,
    "max_tokens": 500,
    "min_tokens": 500,
    "n_choices": null,
    "seed": 0,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": 0.0,
    "top_p": null,
    "top_k": null,
    "extra_args": {}
}
2025-06-18 16:09:27,175 - evalscope - INFO - Test connection successful.
2025-06-18 16:09:28,370 - evalscope - INFO - Save the data base to: outputs/20250618_160921/Qwen2.5-32B-Instruct-GPTQ-Int8/benchmark_data.db
Processing: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:24<00:00, 24.28s/it]
2025-06-18 16:09:52,686 - evalscope - INFO - 
Benchmarking summary:
+-----------------------------------+------------+
| Key                               |      Value |
+===================================+============+
| Time taken for tests (s)          |    24.0493 |
+-----------------------------------+------------+
| Number of concurrency             |     1      |
+-----------------------------------+------------+
| Total requests                    |     1      |
+-----------------------------------+------------+
| Succeed requests                  |     1      |
+-----------------------------------+------------+
| Failed requests                   |     0      |
+-----------------------------------+------------+
| Output token throughput (tok/s)   |    20.7907 |
+-----------------------------------+------------+
| Total token throughput (tok/s)    |  1334.72   |
+-----------------------------------+------------+
| Request throughput (req/s)        |     0.0416 |
+-----------------------------------+------------+
| Average latency (s)               |    24.0486 |
+-----------------------------------+------------+
| Average time to first token (s)   |     0.26   |
+-----------------------------------+------------+
| Average time per output token (s) |     0.0476 |
+-----------------------------------+------------+
| Average input tokens per request  | 31599      |
+-----------------------------------+------------+
| Average output tokens per request |   500      |
+-----------------------------------+------------+
| Average package latency (s)       |     0.0476 |
+-----------------------------------+------------+
| Average package per request       |   500      |
+-----------------------------------+------------+
2025-06-18 16:09:52,688 - evalscope - INFO - 
Percentile results:
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| Percentiles | TTFT (s) | ITL (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Output (tok/s) | Total (tok/s) |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
|     10%     |   0.26   | 0.0475  |  0.0476  |   24.0486   |    31599     |      500      |    20.7913     |   1334.7573   |
|     25%     |   0.26   | 0.0476  |  0.0476  |   24.0486   |    31599     |      500      |    20.7913     |   1334.7573   |
|     50%     |   0.26   | 0.0477  |  0.0476  |   24.0486   |    31599     |      500      |    20.7913     |   1334.7573   |
|     66%     |   0.26   | 0.0477  |  0.0476  |   24.0486   |    31599     |      500      |    20.7913     |   1334.7573   |
|     75%     |   0.26   | 0.0477  |  0.0476  |   24.0486   |    31599     |      500      |    20.7913     |   1334.7573   |
|     80%     |   0.26   | 0.0477  |  0.0476  |   24.0486   |    31599     |      500      |    20.7913     |   1334.7573   |
|     90%     |   0.26   | 0.0478  |  0.0476  |   24.0486   |    31599     |      500      |    20.7913     |   1334.7573   |
|     95%     |   0.26   | 0.0479  |  0.0476  |   24.0486   |    31599     |      500      |    20.7913     |   1334.7573   |
|     98%     |   0.26   |  0.048  |  0.0476  |   24.0486   |    31599     |      500      |    20.7913     |   1334.7573   |
|     99%     |   0.26   |  0.048  |  0.0476  |   24.0486   |    31599     |      500      |    20.7913     |   1334.7573   |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
2025-06-18 16:09:52,688 - evalscope - INFO - Save the summary to: outputs/20250618_160921/Qwen2.5-32B-Instruct-GPTQ-Int8
swanlab: Experiment Qwen2.5-32B-Instruct-GPTQ-Int8_20250618_160921 has completed
swanlab: 🌟 Run `swanlab watch /home/zmy/llm_test/outputs` to view SwanLab Experiment Dashboard locally
2025-06-18 16:09:56,706 - evalscope - INFO - Save the result to: outputs/20250618_160956/Qwen2.5-32B-Instruct-GPTQ-Int8
swanlab: Tracking run with swanlab version 0.6.1
swanlab: Run data will be saved locally in /home/zmy/llm_test/outputs/run-20250618_161001-7a024204
swanlab: 🌟 Run `swanlab watch /home/zmy/llm_test/outputs` to view SwanLab Experiment Dashboard locally
2025-06-18 16:10:02,310 - evalscope - INFO - Starting benchmark with args: 
2025-06-18 16:10:02,310 - evalscope - INFO - {
    "model": "Qwen/Qwen2.5-32B-Instruct-GPTQ-Int8",
    "model_id": "Qwen2.5-32B-Instruct-GPTQ-Int8",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": "models/Qwen2.5-32B-Instruct-GPTQ-Int8",
    "port": 8877,
    "url": "http://192.168.0.56:8000/v1/chat/completions",
    "headers": {
        "Authorization": "Bearer EMPTY"
    },
    "connect_timeout": 600,
    "read_timeout": 600,
    "api_key": "EMPTY",
    "no_test_connection": false,
    "number": 1,
    "parallel": 1,
    "rate": -1,
    "log_every_n_query": 10,
    "debug": false,
    "wandb_api_key": null,
    "swanlab_api_key": "local",
    "name": null,
    "outputs_dir": "outputs/20250618_160956/Qwen2.5-32B-Instruct-GPTQ-Int8",
    "max_prompt_length": 30000,
    "min_prompt_length": 30000,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": true,
    "dataset": "random",
    "dataset_path": null,
    "frequency_penalty": null,
    "logprobs": null,
    "max_tokens": 500,
    "min_tokens": 500,
    "n_choices": null,
    "seed": 0,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": 0.0,
    "top_p": null,
    "top_k": null,
    "extra_args": {}
}
2025-06-18 16:10:02,367 - evalscope - INFO - Test connection successful.
2025-06-18 16:10:03,552 - evalscope - INFO - Save the data base to: outputs/20250618_160956/Qwen2.5-32B-Instruct-GPTQ-Int8/benchmark_data.db
Processing: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:24<00:00, 24.30s/it]
2025-06-18 16:10:27,889 - evalscope - INFO - 
Benchmarking summary:
+-----------------------------------+------------+
| Key                               |      Value |
+===================================+============+
| Time taken for tests (s)          |    24.0477 |
+-----------------------------------+------------+
| Number of concurrency             |     1      |
+-----------------------------------+------------+
| Total requests                    |     1      |
+-----------------------------------+------------+
| Succeed requests                  |     1      |
+-----------------------------------+------------+
| Failed requests                   |     0      |
+-----------------------------------+------------+
| Output token throughput (tok/s)   |    20.792  |
+-----------------------------------+------------+
| Total token throughput (tok/s)    |  1334.8    |
+-----------------------------------+------------+
| Request throughput (req/s)        |     0.0416 |
+-----------------------------------+------------+
| Average latency (s)               |    24.047  |
+-----------------------------------+------------+
| Average time to first token (s)   |     0.2594 |
+-----------------------------------+------------+
| Average time per output token (s) |     0.0476 |
+-----------------------------------+------------+
| Average input tokens per request  | 31599      |
+-----------------------------------+------------+
| Average output tokens per request |   500      |
+-----------------------------------+------------+
| Average package latency (s)       |     0.0476 |
+-----------------------------------+------------+
| Average package per request       |   500      |
+-----------------------------------+------------+
2025-06-18 16:10:27,891 - evalscope - INFO - 
Percentile results:
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| Percentiles | TTFT (s) | ITL (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Output (tok/s) | Total (tok/s) |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
|     10%     |  0.2594  | 0.0475  |  0.0476  |   24.047    |    31599     |      500      |    20.7926     |   1334.8428   |
|     25%     |  0.2594  | 0.0476  |  0.0476  |   24.047    |    31599     |      500      |    20.7926     |   1334.8428   |
|     50%     |  0.2594  | 0.0477  |  0.0476  |   24.047    |    31599     |      500      |    20.7926     |   1334.8428   |
|     66%     |  0.2594  | 0.0477  |  0.0476  |   24.047    |    31599     |      500      |    20.7926     |   1334.8428   |
|     75%     |  0.2594  | 0.0477  |  0.0476  |   24.047    |    31599     |      500      |    20.7926     |   1334.8428   |
|     80%     |  0.2594  | 0.0478  |  0.0476  |   24.047    |    31599     |      500      |    20.7926     |   1334.8428   |
|     90%     |  0.2594  | 0.0478  |  0.0476  |   24.047    |    31599     |      500      |    20.7926     |   1334.8428   |
|     95%     |  0.2594  | 0.0479  |  0.0476  |   24.047    |    31599     |      500      |    20.7926     |   1334.8428   |
|     98%     |  0.2594  |  0.048  |  0.0476  |   24.047    |    31599     |      500      |    20.7926     |   1334.8428   |
|     99%     |  0.2594  | 0.0481  |  0.0476  |   24.047    |    31599     |      500      |    20.7926     |   1334.8428   |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
2025-06-18 16:10:27,892 - evalscope - INFO - Save the summary to: outputs/20250618_160956/Qwen2.5-32B-Instruct-GPTQ-Int8
swanlab: Experiment Qwen2.5-32B-Instruct-GPTQ-Int8_20250618_160956 has completed
swanlab: 🌟 Run `swanlab watch /home/zmy/llm_test/outputs` to view SwanLab Experiment Dashboard locally
2025-06-18 16:10:31,961 - evalscope - INFO - Save the result to: outputs/20250618_161031/Qwen2.5-32B-Instruct-GPTQ-Int8
swanlab: Tracking run with swanlab version 0.6.1
swanlab: Run data will be saved locally in /home/zmy/llm_test/outputs/run-20250618_161037-7a024204
swanlab: 🌟 Run `swanlab watch /home/zmy/llm_test/outputs` to view SwanLab Experiment Dashboard locally
2025-06-18 16:10:37,691 - evalscope - INFO - Starting benchmark with args: 
2025-06-18 16:10:37,691 - evalscope - INFO - {
    "model": "Qwen/Qwen2.5-32B-Instruct-GPTQ-Int8",
    "model_id": "Qwen2.5-32B-Instruct-GPTQ-Int8",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": "models/Qwen2.5-32B-Instruct-GPTQ-Int8",
    "port": 8877,
    "url": "http://192.168.0.56:8000/v1/chat/completions",
    "headers": {
        "Authorization": "Bearer EMPTY"
    },
    "connect_timeout": 600,
    "read_timeout": 600,
    "api_key": "EMPTY",
    "no_test_connection": false,
    "number": 1,
    "parallel": 1,
    "rate": -1,
    "log_every_n_query": 10,
    "debug": false,
    "wandb_api_key": null,
    "swanlab_api_key": "local",
    "name": null,
    "outputs_dir": "outputs/20250618_161031/Qwen2.5-32B-Instruct-GPTQ-Int8",
    "max_prompt_length": 30000,
    "min_prompt_length": 30000,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": true,
    "dataset": "random",
    "dataset_path": null,
    "frequency_penalty": null,
    "logprobs": null,
    "max_tokens": 500,
    "min_tokens": 500,
    "n_choices": null,
    "seed": 0,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": 0.0,
    "top_p": null,
    "top_k": null,
    "extra_args": {}
}
2025-06-18 16:10:37,747 - evalscope - INFO - Test connection successful.
2025-06-18 16:10:38,935 - evalscope - INFO - Save the data base to: outputs/20250618_161031/Qwen2.5-32B-Instruct-GPTQ-Int8/benchmark_data.db
Processing: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:24<00:00, 24.25s/it]
2025-06-18 16:11:03,220 - evalscope - INFO - 
Benchmarking summary:
+-----------------------------------+------------+
| Key                               |      Value |
+===================================+============+
| Time taken for tests (s)          |    24.0488 |
+-----------------------------------+------------+
| Number of concurrency             |     1      |
+-----------------------------------+------------+
| Total requests                    |     1      |
+-----------------------------------+------------+
| Succeed requests                  |     1      |
+-----------------------------------+------------+
| Failed requests                   |     0      |
+-----------------------------------+------------+
| Output token throughput (tok/s)   |    20.791  |
+-----------------------------------+------------+
| Total token throughput (tok/s)    |  1334.74   |
+-----------------------------------+------------+
| Request throughput (req/s)        |     0.0416 |
+-----------------------------------+------------+
| Average latency (s)               |    24.0482 |
+-----------------------------------+------------+
| Average time to first token (s)   |     0.2585 |
+-----------------------------------+------------+
| Average time per output token (s) |     0.0476 |
+-----------------------------------+------------+
| Average input tokens per request  | 31599      |
+-----------------------------------+------------+
| Average output tokens per request |   500      |
+-----------------------------------+------------+
| Average package latency (s)       |     0.0476 |
+-----------------------------------+------------+
| Average package per request       |   500      |
+-----------------------------------+------------+
2025-06-18 16:11:03,221 - evalscope - INFO - 
Percentile results:
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| Percentiles | TTFT (s) | ITL (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Output (tok/s) | Total (tok/s) |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
|     10%     |  0.2585  | 0.0475  |  0.0476  |   24.0482   |    31599     |      500      |    20.7916     |   1334.7796   |
|     25%     |  0.2585  | 0.0476  |  0.0476  |   24.0482   |    31599     |      500      |    20.7916     |   1334.7796   |
|     50%     |  0.2585  | 0.0477  |  0.0476  |   24.0482   |    31599     |      500      |    20.7916     |   1334.7796   |
|     66%     |  0.2585  | 0.0477  |  0.0476  |   24.0482   |    31599     |      500      |    20.7916     |   1334.7796   |
|     75%     |  0.2585  | 0.0477  |  0.0476  |   24.0482   |    31599     |      500      |    20.7916     |   1334.7796   |
|     80%     |  0.2585  | 0.0477  |  0.0476  |   24.0482   |    31599     |      500      |    20.7916     |   1334.7796   |
|     90%     |  0.2585  | 0.0478  |  0.0476  |   24.0482   |    31599     |      500      |    20.7916     |   1334.7796   |
|     95%     |  0.2585  | 0.0479  |  0.0476  |   24.0482   |    31599     |      500      |    20.7916     |   1334.7796   |
|     98%     |  0.2585  |  0.048  |  0.0476  |   24.0482   |    31599     |      500      |    20.7916     |   1334.7796   |
|     99%     |  0.2585  | 0.0481  |  0.0476  |   24.0482   |    31599     |      500      |    20.7916     |   1334.7796   |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
2025-06-18 16:11:03,222 - evalscope - INFO - Save the summary to: outputs/20250618_161031/Qwen2.5-32B-Instruct-GPTQ-Int8
swanlab: Experiment Qwen2.5-32B-Instruct-GPTQ-Int8_20250618_161032 has completed
swanlab: 🌟 Run `swanlab watch /home/zmy/llm_test/outputs` to view SwanLab Experiment Dashboard locally
2025-06-18 16:11:07,277 - evalscope - INFO - Save the result to: outputs/20250618_161107/Qwen2.5-32B-Instruct-GPTQ-Int8
swanlab: Tracking run with swanlab version 0.6.1
swanlab: Run data will be saved locally in /home/zmy/llm_test/outputs/run-20250618_161112-7a024204
swanlab: 🌟 Run `swanlab watch /home/zmy/llm_test/outputs` to view SwanLab Experiment Dashboard locally
2025-06-18 16:11:13,015 - evalscope - INFO - Starting benchmark with args: 
2025-06-18 16:11:13,015 - evalscope - INFO - {
    "model": "Qwen/Qwen2.5-32B-Instruct-GPTQ-Int8",
    "model_id": "Qwen2.5-32B-Instruct-GPTQ-Int8",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": "models/Qwen2.5-32B-Instruct-GPTQ-Int8",
    "port": 8877,
    "url": "http://192.168.0.56:8000/v1/chat/completions",
    "headers": {
        "Authorization": "Bearer EMPTY"
    },
    "connect_timeout": 600,
    "read_timeout": 600,
    "api_key": "EMPTY",
    "no_test_connection": false,
    "number": 1,
    "parallel": 1,
    "rate": -1,
    "log_every_n_query": 10,
    "debug": false,
    "wandb_api_key": null,
    "swanlab_api_key": "local",
    "name": null,
    "outputs_dir": "outputs/20250618_161107/Qwen2.5-32B-Instruct-GPTQ-Int8",
    "max_prompt_length": 30000,
    "min_prompt_length": 30000,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": true,
    "dataset": "random",
    "dataset_path": null,
    "frequency_penalty": null,
    "logprobs": null,
    "max_tokens": 500,
    "min_tokens": 500,
    "n_choices": null,
    "seed": 0,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": 0.0,
    "top_p": null,
    "top_k": null,
    "extra_args": {}
}
2025-06-18 16:11:13,071 - evalscope - INFO - Test connection successful.
2025-06-18 16:11:14,292 - evalscope - INFO - Save the data base to: outputs/20250618_161107/Qwen2.5-32B-Instruct-GPTQ-Int8/benchmark_data.db
Processing: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:24<00:00, 24.30s/it]
2025-06-18 16:11:38,622 - evalscope - INFO - 
Benchmarking summary:
+-----------------------------------+------------+
| Key                               |      Value |
+===================================+============+
| Time taken for tests (s)          |    24.0524 |
+-----------------------------------+------------+
| Number of concurrency             |     1      |
+-----------------------------------+------------+
| Total requests                    |     1      |
+-----------------------------------+------------+
| Succeed requests                  |     1      |
+-----------------------------------+------------+
| Failed requests                   |     0      |
+-----------------------------------+------------+
| Output token throughput (tok/s)   |    20.788  |
+-----------------------------------+------------+
| Total token throughput (tok/s)    |  1334.55   |
+-----------------------------------+------------+
| Request throughput (req/s)        |     0.0416 |
+-----------------------------------+------------+
| Average latency (s)               |    24.0518 |
+-----------------------------------+------------+
| Average time to first token (s)   |     0.2585 |
+-----------------------------------+------------+
| Average time per output token (s) |     0.0476 |
+-----------------------------------+------------+
| Average input tokens per request  | 31599      |
+-----------------------------------+------------+
| Average output tokens per request |   500      |
+-----------------------------------+------------+
| Average package latency (s)       |     0.0476 |
+-----------------------------------+------------+
| Average package per request       |   500      |
+-----------------------------------+------------+
2025-06-18 16:11:38,624 - evalscope - INFO - 
Percentile results:
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| Percentiles | TTFT (s) | ITL (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Output (tok/s) | Total (tok/s) |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
|     10%     |  0.2585  | 0.0475  |  0.0476  |   24.0518   |    31599     |      500      |    20.7885     |   1334.5786   |
|     25%     |  0.2585  | 0.0476  |  0.0476  |   24.0518   |    31599     |      500      |    20.7885     |   1334.5786   |
|     50%     |  0.2585  | 0.0477  |  0.0476  |   24.0518   |    31599     |      500      |    20.7885     |   1334.5786   |
|     66%     |  0.2585  | 0.0477  |  0.0476  |   24.0518   |    31599     |      500      |    20.7885     |   1334.5786   |
|     75%     |  0.2585  | 0.0477  |  0.0476  |   24.0518   |    31599     |      500      |    20.7885     |   1334.5786   |
|     80%     |  0.2585  | 0.0477  |  0.0476  |   24.0518   |    31599     |      500      |    20.7885     |   1334.5786   |
|     90%     |  0.2585  | 0.0478  |  0.0476  |   24.0518   |    31599     |      500      |    20.7885     |   1334.5786   |
|     95%     |  0.2585  | 0.0479  |  0.0476  |   24.0518   |    31599     |      500      |    20.7885     |   1334.5786   |
|     98%     |  0.2585  |  0.048  |  0.0476  |   24.0518   |    31599     |      500      |    20.7885     |   1334.5786   |
|     99%     |  0.2585  |  0.048  |  0.0476  |   24.0518   |    31599     |      500      |    20.7885     |   1334.5786   |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
2025-06-18 16:11:38,624 - evalscope - INFO - Save the summary to: outputs/20250618_161107/Qwen2.5-32B-Instruct-GPTQ-Int8
swanlab: Experiment Qwen2.5-32B-Instruct-GPTQ-Int8_20250618_161107 has completed
swanlab: 🌟 Run `swanlab watch /home/zmy/llm_test/outputs` to view SwanLab Experiment Dashboard locally
```

--parallel 1 + --number 5 是没有间隔的进行请求，一个请求结束后立马发送下一个请求。你写的脚本的形式，即使是看起来连续，但中间还是有间隔。

> --parallel 1 + --number 5 是没有间隔的进行请求，一个请求结束后立马发送下一个请求。你写的脚本的形式，即使是看起来连续，但中间还是有间隔。

好的，明白了，感谢！

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！