Repository: modelscope/evalscope
Issue: 性能测试时，指定了输入token长度，但在结果中显示的输入token长度为0
Root cause: Validation Gap

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [x] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [x] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [x] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述

性能测试时，指定了输入token长度，但在结果中显示的输入token长度为0

## EvalScope 版本（必填）
v2.0.0

## 使用的工具

- [x] Perf / 模型推理压测工具


## 执行的代码或指令

请提供您执行的主要代码或指令。

## 错误日志

<img width="827" height="866" alt="Image" src="https://github.com/user-attachments/assets/200e0c64-dad0-4a22-b29d-1987f4659a2f" />

## 运行环境

- 操作系统：
- Python版本：

## 其他信息

如果有其他相关信息，请在此处提供。


请提供完整的评测命令

> 请提供完整的评测命令

nohup evalscope perf \
  --parallel 1 2 4 8 10 \
  --number 50 100 200 400 500  \
  --model DeepSeek-R1 \
  --url http://xxxxxxxxxx:xx/v1/chat/completions \
  --api-key xxxxxxxxxxxxxxxxxxxxxx \
  --api openai \
  --dataset random \
  --max-tokens 1024 \
  --min-tokens 1024 \
  --prefix-length 0 \
  --min-prompt-length 1024 \
  --max-prompt-length 1024 \
  --temperature 0.6 \
  --top-p 0.95 \
  --stream \
  --tokenizer-path /home/ubuntu/test/evalscope_test/config_qwen \
  --extra-args '{"ignore_eos": true}' \
  --swanlab-api-key 'xxxxxxxxxxxxxxxxxxxxx' \
  --name "./outputs/$(date +%Y%m%d_%H%M%S)" \

最后打印的表格是正常的吗

> 最后打印的表格是正常的吗

表格中Input Tokens不正常  

<img width="1109" height="731" alt="Image" src="https://github.com/user-attachments/assets/1a8ca383-2e48-4613-acf6-4c91fd5b1790" />

你使用的模型服务是标准的openai api格式吗，是vllm等框架启动的吗？我这边复现不了。

@Yunnglin 您好，我也遇到了相同的问题，希望能够获得您的帮助。
问题描述：模型通过ollama运行在机器A上，我正在机器B上进行访问，机器A和机器B通过网线直连，这个问题经常会出现，以下是我的命令。
### 正常的情况
**当我的接口访问的是/v1/chat/completions，并且dataset是openqa的时候，这时是正常的。
(evalscope) henry@U20:~$ evalscope perf   --url "http://192.168.137.1:11434/v1/chat/completions"   --model "deepseek-r1:1.5b"   --parallel 10 20    --number 10 20   --max-prompt-length 1024   --dataset "openqa"     --temperature 0.7   --top-p 0.9   --max-tokens 2048**
2025-08-26 13:40:16 - evalscope - INFO: Save the result to: outputs/20250826_134016/deepseek-r1_1.5b
2025-08-26 13:40:16 - evalscope - INFO: Starting benchmark with args:
2025-08-26 13:40:16 - evalscope - INFO: {
    "model": "deepseek-r1:1.5b",
    "model_id": "deepseek-r1:1.5b",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": null,
    "port": 8877,
    "url": "http://192.168.137.1:11434/v1/chat/completions",
    "headers": {},
    "connect_timeout": 600,
    "read_timeout": 600,
    "api_key": null,
    "no_test_connection": false,
    "number": 10,
    "parallel": 10,
    "rate": -1,
    "sleep_interval": 5,
    "log_every_n_query": 10,
    "debug": false,
    "wandb_api_key": null,
    "swanlab_api_key": null,
    "name": null,
    "outputs_dir": "outputs/20250826_134016/deepseek-r1_1.5b/parallel_10_number_10",
    "max_prompt_length": 1024,
    "min_prompt_length": 0,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": true,
    "image_width": 224,
    "image_height": 224,
    "image_format": "RGB",
    "image_num": 1,
    "dataset": "openqa",
    "dataset_path": null,
    "frequency_penalty": null,
    "repetition_penalty": null,
    "logprobs": null,
    "max_tokens": 2048,
    "min_tokens": null,
    "n_choices": null,
    "seed": 0,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": 0.7,
    "top_p": 0.9,
    "top_k": null,
    "extra_args": {}
}
2025-08-26 13:40:20 - evalscope - INFO: Test connection successful.
Downloading Dataset to directory: /home/henry/.cache/modelscope/hub/datasets/AI-ModelScope/HC3-Chinese
2025-08-26 13:40:32,884 - modelscope - INFO - Fetching dataset repo file list...
2025-08-26 13:40:33 - evalscope - INFO: Save the data base to: outputs/20250826_134016/deepseek-r1_1.5b/parallel_10_number_10/benchmark_data.db
Processing:  90%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████               | 9/10 [00:36<00:03,  3.72s/it]2025-08-26 13:41:18 - evalscope - INFO: {
  "Time taken for tests (s)": 45.4623,
  "Number of concurrency": 10,
  "Total requests": 10,
  "Succeed requests": 10,
  "Failed requests": 0,
  "Output token throughput (tok/s)": 121.3532,
  "Total token throughput (tok/s)": 126.6103,
  "Request throughput (req/s)": 0.22,
  "Average latency (s)": 22.8403,
  "Average time to first token (s)": 18.3282,
  "Average time per output token (s)": 0.0081,
  "Average inter-token latency (s)": 0.0082,
  "Average input tokens per request": 23.9,
  "Average output tokens per request": 551.7
}
Processing: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:45<00:00,  4.55s/it]
2025-08-26 13:41:19 - evalscope - INFO:
Benchmarking summary:
+-----------------------------------+----------+
| Key                               |    Value |
+===================================+==========+
| Time taken for tests (s)          |  45.4623 |
+-----------------------------------+----------+
| Number of concurrency             |  10      |
+-----------------------------------+----------+
| Total requests                    |  10      |
+-----------------------------------+----------+
| Succeed requests                  |  10      |
+-----------------------------------+----------+
| Failed requests                   |   0      |
+-----------------------------------+----------+
| Output token throughput (tok/s)   | 121.353  |
+-----------------------------------+----------+
| Total token throughput (tok/s)    | 126.61   |
+-----------------------------------+----------+
| Request throughput (req/s)        |   0.22   |
+-----------------------------------+----------+
| Average latency (s)               |  22.8403 |
+-----------------------------------+----------+
| Average time to first token (s)   |  18.3282 |
+-----------------------------------+----------+
| Average time per output token (s) |   0.0081 |
+-----------------------------------+----------+
| Average inter-token latency (s)   |   0.0082 |
+-----------------------------------+----------+
| Average input tokens per request  |  23.9    |
+-----------------------------------+----------+
| Average output tokens per request | 551.7    |
+-----------------------------------+----------+
2025-08-26 13:41:19 - evalscope - INFO:
Percentile results:
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| Percentiles | TTFT (s) | ITL (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Output (tok/s) | Total (tok/s) |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
|     10%     |  4.6965  | 0.0079  |  0.008   |   8.2347    |      15      |      222      |     6.701      |    7.4255     |
|     25%     |  8.2496  |  0.008  |  0.0081  |   11.8084   |      18      |      367      |    10.1652     |    11.1624    |
|     50%     | 20.0788  | 0.0082  |  0.0082  |   25.5592   |      24      |      553      |    26.2919     |    27.3092    |
|     66%     | 25.5785  | 0.0083  |  0.0082  |   31.3484   |      26      |      672      |    37.5155     |    38.7858    |
|     75%     | 31.3666  | 0.0084  |  0.0082  |   33.1292   |      27      |      705      |     48.354     |    49.4507    |
|     80%     | 33.1545  | 0.0084  |  0.0083  |   36.1035   |      32      |      970      |    53.4322     |    55.2537    |
|     90%     |  36.119  | 0.0085  |  0.0084  |   45.4633   |      36      |     1120      |    118.1052    |   124.9395    |
|     95%     |  36.119  | 0.0087  |  0.0084  |   45.4633   |      36      |     1120      |    118.1052    |   124.9395    |
|     98%     |  36.119  | 0.0088  |  0.0084  |   45.4633   |      36      |     1120      |    118.1052    |   124.9395    |
|     99%     |  36.119  | 0.0089  |  0.0084  |   45.4633   |      36      |     1120      |    118.1052    |   124.9395    |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
2025-08-26 13:41:19 - evalscope - INFO: Save the summary to: outputs/20250826_134016/deepseek-r1_1.5b/parallel_10_number_10
2025-08-26 13:41:19 - evalscope - INFO: Sleeping for 5 seconds before the next run...
2025-08-26 13:41:24 - evalscope - INFO: Starting benchmark with args:
2025-08-26 13:41:24 - evalscope - INFO: {
    "model": "deepseek-r1:1.5b",
    "model_id": "deepseek-r1:1.5b",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": null,
    "port": 8877,
    "url": "http://192.168.137.1:11434/v1/chat/completions",
    "headers": {},
    "connect_timeout": 600,
    "read_timeout": 600,
    "api_key": null,
    "no_test_connection": false,
    "number": 20,
    "parallel": 20,
    "rate": -1,
    "sleep_interval": 5,
    "log_every_n_query": 10,
    "debug": false,
    "wandb_api_key": null,
    "swanlab_api_key": null,
    "name": null,
    "outputs_dir": "outputs/20250826_134016/deepseek-r1_1.5b/parallel_20_number_20",
    "max_prompt_length": 1024,
    "min_prompt_length": 0,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": true,
    "image_width": 224,
    "image_height": 224,
    "image_format": "RGB",
    "image_num": 1,
    "dataset": "openqa",
    "dataset_path": "/home/henry/.cache/modelscope/hub/datasets/AI-ModelScope/HC3-Chinese/open_qa.jsonl",
    "frequency_penalty": null,
    "repetition_penalty": null,
    "logprobs": null,
    "max_tokens": 2048,
    "min_tokens": null,
    "n_choices": null,
    "seed": 0,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": 0.7,
    "top_p": 0.9,
    "top_k": null,
    "extra_args": {}
}
2025-08-26 13:41:24 - evalscope - INFO: Test connection successful.
2025-08-26 13:41:24 - evalscope - INFO: Save the data base to: outputs/20250826_134016/deepseek-r1_1.5b/parallel_20_number_20/benchmark_data.db
Processing:  45%|███████████████████████████████████████████████████████████████████▌                                                                                  | 9/20 [00:34<00:52,  4.77s/it]2025-08-26 13:42:05 - evalscope - INFO: {
  "Time taken for tests (s)": 41.8005,
  "Number of concurrency": 20,
  "Total requests": 10,
  "Succeed requests": 10,
  "Failed requests": 0,
  "Output token throughput (tok/s)": 121.6014,
  "Total token throughput (tok/s)": 127.63,
  "Request throughput (req/s)": 0.2392,
  "Average latency (s)": 20.4816,
  "Average time to first token (s)": 16.3379,
  "Average time per output token (s)": 0.0081,
  "Average inter-token latency (s)": 0.0082,
  "Average input tokens per request": 25.2,
  "Average output tokens per request": 508.3
}
Processing:  95%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌       | 19/20 [01:35<00:07,  7.25s/it]2025-08-26 13:43:05 - evalscope - INFO: {
  "Time taken for tests (s)": 100.9229,
  "Number of concurrency": 20,
  "Total requests": 20,
  "Succeed requests": 20,
  "Failed requests": 0,
  "Output token throughput (tok/s)": 121.2213,
  "Total token throughput (tok/s)": 126.0269,
  "Request throughput (req/s)": 0.1982,
  "Average latency (s)": 45.483,
  "Average time to first token (s)": 40.4641,
  "Average time per output token (s)": 0.0082,
  "Average inter-token latency (s)": 0.0083,
  "Average input tokens per request": 24.25,
  "Average output tokens per request": 611.7
}
Processing: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [01:40<00:00,  5.05s/it]
2025-08-26 13:43:05 - evalscope - INFO:
Benchmarking summary:
+-----------------------------------+----------+
| Key                               |    Value |
+===================================+==========+
| Time taken for tests (s)          | 100.923  |
+-----------------------------------+----------+
| Number of concurrency             |  20      |
+-----------------------------------+----------+
| Total requests                    |  20      |
+-----------------------------------+----------+
| Succeed requests                  |  20      |
+-----------------------------------+----------+
| Failed requests                   |   0      |
+-----------------------------------+----------+
| Output token throughput (tok/s)   | 121.221  |
+-----------------------------------+----------+
| Total token throughput (tok/s)    | 126.027  |
+-----------------------------------+----------+
| Request throughput (req/s)        |   0.1982 |
+-----------------------------------+----------+
| Average latency (s)               |  45.483  |
+-----------------------------------+----------+
| Average time to first token (s)   |  40.4641 |
+-----------------------------------+----------+
| Average time per output token (s) |   0.0082 |
+-----------------------------------+----------+
| Average inter-token latency (s)   |   0.0083 |
+-----------------------------------+----------+
| Average input tokens per request  |  24.25   |
+-----------------------------------+----------+
| Average output tokens per request | 611.7    |
+-----------------------------------+----------+
2025-08-26 13:43:05 - evalscope - INFO:
Percentile results:
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| Percentiles | TTFT (s) | ITL (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Output (tok/s) | Total (tok/s) |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
|     10%     | 10.1921  | 0.0079  |  0.008   |   11.954    |      16      |      319      |     6.9856     |    7.2532     |
|     25%     | 15.7436  |  0.008  |  0.0081  |   18.6961   |      21      |      443      |     8.8071     |    9.1053     |
|     50%     | 41.8153  | 0.0082  |  0.0082  |   46.7035   |      23      |      599      |    13.7428     |    14.3026    |
|     66%     | 52.8941  | 0.0083  |  0.0082  |   58.9784   |      26      |      739      |    19.6298     |    20.5788    |
|     75%     | 63.2878  | 0.0084  |  0.0083  |   71.4555   |      29      |      834      |    23.0595     |    24.0149    |
|     80%     | 71.4749  | 0.0084  |  0.0083  |   79.5359   |      32      |      905      |    25.9358     |    26.7669    |
|     90%     | 85.7181  | 0.0086  |  0.0083  |   95.1113   |      36      |      982      |    54.3559     |    57.5013    |
|     95%     | 95.1295  | 0.0087  |  0.0084  |  100.9216   |      40      |     1120      |    118.6198    |   123.2092    |
|     98%     | 95.1295  | 0.0088  |  0.0084  |  100.9216   |      40      |     1120      |    118.6198    |   123.2092    |
|     99%     | 95.1295  | 0.0089  |  0.0084  |  100.9216   |      40      |     1120      |    118.6198    |   123.2092    |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
2025-08-26 13:43:05 - evalscope - INFO: Save the summary to: outputs/20250826_134016/deepseek-r1_1.5b/parallel_20_number_20
╭──────────────────────────────────────────────────────────╮
│ Performance Test Summary Report                          │
╰──────────────────────────────────────────────────────────╯

Basic Information:
┌───────────────────────┬──────────────────────────────────┐
│ Model                 │ deepseek-r1:1.5b                 │
│ Total Generated       │ 17,751.0 tokens                  │
│ Total Test Time       │ 146.39 seconds                   │
│ Avg Output Rate       │ 121.26 tokens/sec                │
└───────────────────────┴──────────────────────────────────┘


                                    Detailed Performance Metrics
┏━━━━━━┳━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┓
┃      ┃      ┃      Avg ┃      P99 ┃    Gen. ┃      Avg ┃     P99 ┃      Avg ┃     P99 ┃   Success┃
┃Conc. ┃  RPS ┃  Lat.(s) ┃  Lat.(s) ┃  toks/s ┃  TTFT(s) ┃ TTFT(s) ┃  TPOT(s) ┃ TPOT(s) ┃      Rate┃
┡━━━━━━╇━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━┩
│   10 │ 0.22 │   22.840 │   45.463 │  121.35 │   18.328 │  36.119 │    0.008 │   0.008 │    100.0%│
│   20 │ 0.20 │   45.483 │  100.922 │  121.22 │   40.464 │  95.129 │    0.008 │   0.008 │    100.0%│
└──────┴──────┴──────────┴──────────┴─────────┴──────────┴─────────┴──────────┴─────────┴──────────┘


               Best Performance Configuration
 Highest RPS         Concurrency 10 (0.22 req/sec)
 Lowest Latency      Concurrency 10 (22.840 seconds)

### 异常的情况
**当我将dataset更换为speed_benchmark时，会得到提示/v1/chat/completions无法使用，应该更换为/v1/completions，按照提示进行更换后，此时运行结果是
(evalscope) henry@U20:~$ evalscope perf   --url "http://192.168.137.1:11434/v1/completions"   --model "deepseek-r1:1.5b"   --parallel 10 20    --number 10 20   --max-prompt-length 1024   --dataset "speed_benchmark"     --temperature 0.7   --top-p 0.9   --max-tokens 2048**
2025-08-26 13:44:59 - evalscope - INFO: Save the result to: outputs/20250826_134459/deepseek-r1_1.5b
2025-08-26 13:44:59 - evalscope - INFO: Starting benchmark with args:
2025-08-26 13:44:59 - evalscope - INFO: {
    "model": "deepseek-r1:1.5b",
    "model_id": "deepseek-r1:1.5b",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": null,
    "port": 8877,
    "url": "http://192.168.137.1:11434/v1/completions",
    "headers": {},
    "connect_timeout": 600,
    "read_timeout": 600,
    "api_key": null,
    "no_test_connection": false,
    "number": 10,
    "parallel": 10,
    "rate": -1,
    "sleep_interval": 5,
    "log_every_n_query": 10,
    "debug": false,
    "wandb_api_key": null,
    "swanlab_api_key": null,
    "name": null,
    "outputs_dir": "outputs/20250826_134459/deepseek-r1_1.5b/parallel_10_number_10",
    "max_prompt_length": 1024,
    "min_prompt_length": 0,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": false,
    "image_width": 224,
    "image_height": 224,
    "image_format": "RGB",
    "image_num": 1,
    "dataset": "speed_benchmark",
    "dataset_path": null,
    "frequency_penalty": null,
    "repetition_penalty": null,
    "logprobs": null,
    "max_tokens": 2048,
    "min_tokens": null,
    "n_choices": null,
    "seed": 0,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": 0.7,
    "top_p": 0.9,
    "top_k": null,
    "extra_args": {}
}
2025-08-26 13:44:59 - evalscope - INFO: Test connection successful.
2025-08-26 13:44:59 - evalscope - INFO: Save the data base to: outputs/20250826_134459/deepseek-r1_1.5b/parallel_10_number_10/benchmark_data.db
Processing:  90%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████               | 9/10 [00:17<00:01,  1.58s/it]2025-08-26 13:45:21 - evalscope - INFO: {
  "Time taken for tests (s)": 21.6414,
  "Number of concurrency": 10,
  "Total requests": 10,
  "Succeed requests": 10,
  "Failed requests": 0,
  "Output token throughput (tok/s)": 14.5092,
  "Total token throughput (tok/s)": 204.3304,
  "Request throughput (req/s)": 0.4621,
  "Average latency (s)": 12.7189,
  "Average time to first token (s)": 12.182,
  "Average time per output token (s)": 0.0034,
  "Average inter-token latency (s)": 0.011,
  "Average input tokens per request": 410.8,
  "Average output tokens per request": 31.4
}
Processing: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:21<00:00,  2.17s/it]
2025-08-26 13:45:21 - evalscope - INFO:
Benchmarking summary:
+-----------------------------------+----------+
| Key                               |    Value |
+===================================+==========+
| Time taken for tests (s)          |  21.6414 |
+-----------------------------------+----------+
| Number of concurrency             |  10      |
+-----------------------------------+----------+
| Total requests                    |  10      |
+-----------------------------------+----------+
| Succeed requests                  |  10      |
+-----------------------------------+----------+
| Failed requests                   |   0      |
+-----------------------------------+----------+
| Output token throughput (tok/s)   |  14.5092 |
+-----------------------------------+----------+
| Total token throughput (tok/s)    | 204.33   |
+-----------------------------------+----------+
| Request throughput (req/s)        |   0.4621 |
+-----------------------------------+----------+
| Average latency (s)               |  12.7189 |
+-----------------------------------+----------+
| Average time to first token (s)   |  12.182  |
+-----------------------------------+----------+
| Average time per output token (s) |   0.0034 |
+-----------------------------------+----------+
| Average inter-token latency (s)   |   0.011  |
+-----------------------------------+----------+
| Average input tokens per request  | 410.8    |
+-----------------------------------+----------+
| Average output tokens per request |  31.4    |
+-----------------------------------+----------+
2025-08-26 13:45:21 - evalscope - INFO:
Percentile results:
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| Percentiles | TTFT (s) | ITL (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Output (tok/s) | Total (tok/s) |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
|     10%     |  8.3743  | 0.0079  |   0.0    |   8.7156    |      0       |       0       |      0.0       |      0.0      |
|     25%     |  8.7363  | 0.0107  |   0.0    |   8.8537    |      0       |       0       |      0.0       |      0.0      |
|     50%     | 12.3534  | 0.0108  |   0.0    |   12.737    |      0       |       0       |      0.0       |      0.0      |
|     66%     | 15.0321  |  0.011  |  0.0078  |   15.4231   |      4       |      16       |     1.7574     |    2.1968     |
|     75%     | 17.2921  | 0.0117  |  0.0078  |   17.6818   |      4       |      16       |     1.7829     |    2.2287     |
|     80%     | 17.7115  | 0.0119  |  0.0078  |   17.9593   |      4       |      16       |     1.8071     |    2.2589     |
|     90%     | 21.2484  |  0.013  |  0.0108  |   21.6396   |     4096     |      266      |    43.6005     |   714.9829    |
|     95%     | 21.2484  | 0.0135  |  0.0108  |   21.6396   |     4096     |      266      |    43.6005     |   714.9829    |
|     98%     | 21.2484  | 0.0214  |  0.0108  |   21.6396   |     4096     |      266      |    43.6005     |   714.9829    |
|     99%     | 21.2484  | 0.0215  |  0.0108  |   21.6396   |     4096     |      266      |    43.6005     |   714.9829    |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
2025-08-26 13:45:21 - evalscope - INFO:
Speed Benchmark Results:
+---------------+-----------------+----------------+
| Prompt Tokens | Speed(tokens/s) | GPU Memory(GB) |
+---------------+-----------------+----------------+
|       0       |       0.0       |      0.0       |
|       4       |      1.78       |      0.0       |
|     4096      |      43.6       |      0.0       |
+---------------+-----------------+----------------+
2025-08-26 13:45:21 - evalscope - INFO: Save the summary to: outputs/20250826_134459/deepseek-r1_1.5b/parallel_10_number_10
2025-08-26 13:45:21 - evalscope - INFO: Sleeping for 5 seconds before the next run...
2025-08-26 13:45:26 - evalscope - INFO: Starting benchmark with args:
2025-08-26 13:45:26 - evalscope - INFO: {
    "model": "deepseek-r1:1.5b",
    "model_id": "deepseek-r1:1.5b",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": null,
    "port": 8877,
    "url": "http://192.168.137.1:11434/v1/completions",
    "headers": {},
    "connect_timeout": 600,
    "read_timeout": 600,
    "api_key": null,
    "no_test_connection": false,
    "number": 20,
    "parallel": 20,
    "rate": -1,
    "sleep_interval": 5,
    "log_every_n_query": 10,
    "debug": false,
    "wandb_api_key": null,
    "swanlab_api_key": null,
    "name": null,
    "outputs_dir": "outputs/20250826_134459/deepseek-r1_1.5b/parallel_20_number_20",
    "max_prompt_length": 1024,
    "min_prompt_length": 0,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": false,
    "image_width": 224,
    "image_height": 224,
    "image_format": "RGB",
    "image_num": 1,
    "dataset": "speed_benchmark",
    "dataset_path": null,
    "frequency_penalty": null,
    "repetition_penalty": null,
    "logprobs": null,
    "max_tokens": 2048,
    "min_tokens": null,
    "n_choices": null,
    "seed": 0,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": 0.7,
    "top_p": 0.9,
    "top_k": null,
    "extra_args": {}
}
2025-08-26 13:45:26 - evalscope - INFO: Test connection successful.
2025-08-26 13:45:26 - evalscope - INFO: Save the data base to: outputs/20250826_134459/deepseek-r1_1.5b/parallel_20_number_20/benchmark_data.db
Processing:  45%|███████████████████████████████████████████████████████████████████▌                                                                                  | 9/20 [00:15<00:22,  2.04s/it]2025-08-26 13:45:44 - evalscope - INFO: {
  "Time taken for tests (s)": 17.3469,
  "Number of concurrency": 20,
  "Total requests": 10,
  "Succeed requests": 10,
  "Failed requests": 0,
  "Output token throughput (tok/s)": 0.0,
  "Total token throughput (tok/s)": 0.0,
  "Request throughput (req/s)": 0.5765,
  "Average latency (s)": 8.615,
  "Average time to first token (s)": 8.2971,
  "Average time per output token (s)": 0.0,
  "Average inter-token latency (s)": 0.0103,
  "Average input tokens per request": 0.0,
  "Average output tokens per request": 0.0
}
Processing:  95%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌       | 19/20 [00:39<00:02,  2.49s/it]2025-08-26 13:46:08 - evalscope - INFO: {
  "Time taken for tests (s)": 41.3032,
  "Number of concurrency": 20,
  "Total requests": 20,
  "Succeed requests": 20,
  "Failed requests": 0,
  "Output token throughput (tok/s)": 0.0,
  "Total token throughput (tok/s)": 0.0,
  "Request throughput (req/s)": 0.4842,
  "Average latency (s)": 19.5283,
  "Average time to first token (s)": 19.1755,
  "Average time per output token (s)": 0.0,
  "Average inter-token latency (s)": 0.0114,
  "Average input tokens per request": 0.0,
  "Average output tokens per request": 0.0
}
Processing: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:41<00:00,  2.07s/it]
2025-08-26 13:46:08 - evalscope - INFO:
Benchmarking summary:
+-----------------------------------+---------+
| Key                               |   Value |
+===================================+=========+
| Time taken for tests (s)          | 41.3032 |
+-----------------------------------+---------+
| Number of concurrency             | 20      |
+-----------------------------------+---------+
| Total requests                    | 20      |
+-----------------------------------+---------+
| Succeed requests                  | 20      |
+-----------------------------------+---------+
| Failed requests                   |  0      |
+-----------------------------------+---------+
| Output token throughput (tok/s)   |  0      |
+-----------------------------------+---------+
| Total token throughput (tok/s)    |  0      |
+-----------------------------------+---------+
| Request throughput (req/s)        |  0.4842 |
+-----------------------------------+---------+
| Average latency (s)               | 19.5283 |
+-----------------------------------+---------+
| Average time to first token (s)   | 19.1755 |
+-----------------------------------+---------+
| Average time per output token (s) |  0      |
+-----------------------------------+---------+
| Average inter-token latency (s)   |  0.0114 |
+-----------------------------------+---------+
| Average input tokens per request  |  0      |
+-----------------------------------+---------+
| Average output tokens per request |  0      |
+-----------------------------------+---------+
2025-08-26 13:46:08 - evalscope - INFO:
Percentile results:
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
| Percentiles | TTFT (s) | ITL (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Output (tok/s) | Total (tok/s) |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
|     10%     |  3.9203  | 0.0079  |   0.0    |   4.1654    |      0       |       0       |      0.0       |      0.0      |
|     25%     |  8.1277  | 0.0082  |   0.0    |   8.3772    |      0       |       0       |      0.0       |      0.0      |
|     50%     | 19.6369  | 0.0122  |   0.0    |   20.0387   |      0       |       0       |      0.0       |      0.0      |
|     66%     | 26.9762  | 0.0127  |   0.0    |   27.3733   |      0       |       0       |      0.0       |      0.0      |
|     75%     | 30.9195  | 0.0131  |   0.0    |   31.3193   |      0       |       0       |      0.0       |      0.0      |
|     80%     | 33.6285  | 0.0132  |   0.0    |   34.0377   |      0       |       0       |      0.0       |      0.0      |
|     90%     | 38.6297  | 0.0134  |   0.0    |   39.0351   |      0       |       0       |      0.0       |      0.0      |
|     95%     | 40.8989  | 0.0135  |   0.0    |   41.3009   |      0       |       0       |      0.0       |      0.0      |
|     98%     | 40.8989  | 0.0189  |   0.0    |   41.3009   |      0       |       0       |      0.0       |      0.0      |
|     99%     | 40.8989  | 0.0194  |   0.0    |   41.3009   |      0       |       0       |      0.0       |      0.0      |
+-------------+----------+---------+----------+-------------+--------------+---------------+----------------+---------------+
2025-08-26 13:46:08 - evalscope - INFO:
Speed Benchmark Results:
+---------------+-----------------+----------------+
| Prompt Tokens | Speed(tokens/s) | GPU Memory(GB) |
+---------------+-----------------+----------------+
|       0       |       0.0       |      0.0       |
+---------------+-----------------+----------------+
2025-08-26 13:46:08 - evalscope - INFO: Save the summary to: outputs/20250826_134459/deepseek-r1_1.5b/parallel_20_number_20
╭──────────────────────────────────────────────────────────╮
│ Performance Test Summary Report                          │
╰──────────────────────────────────────────────────────────╯

Basic Information:
┌───────────────────────┬──────────────────────────────────┐
│ Model                 │ deepseek-r1:1.5b                 │
│ Total Generated       │ 314.0 tokens                     │
│ Total Test Time       │ 62.94 seconds                    │
│ Avg Output Rate       │ 4.99 tokens/sec                  │
└───────────────────────┴──────────────────────────────────┘


                                    Detailed Performance Metrics
┏━━━━━━┳━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┓
┃      ┃      ┃      Avg ┃      P99 ┃    Gen. ┃      Avg ┃     P99 ┃      Avg ┃     P99 ┃   Success┃
┃Conc. ┃  RPS ┃  Lat.(s) ┃  Lat.(s) ┃  toks/s ┃  TTFT(s) ┃ TTFT(s) ┃  TPOT(s) ┃ TPOT(s) ┃      Rate┃
┡━━━━━━╇━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━┩
│   10 │ 0.46 │   12.719 │   21.640 │   14.51 │   12.182 │  21.248 │    0.003 │   0.011 │    100.0%│
│   20 │ 0.48 │   19.528 │   41.301 │    0.00 │   19.175 │  40.899 │    0.000 │   0.000 │    100.0%│
└──────┴──────┴──────────┴──────────┴─────────┴──────────┴─────────┴──────────┴─────────┴──────────┘


               Best Performance Configuration
 Highest RPS         Concurrency 20 (0.48 req/sec)
 Lowest Latency      Concurrency 10 (12.719 seconds)




main分支已修复该问题，可以再试一下

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！