Repository: modelscope/evalscope
Issue: Average QPS = -1
Root cause: Validation Gap

## 问题描述 / Issue Description

我对在笔记本上的LM Studio上运行的 qwen2.5-coder-14b-instruct 进行检查压测，发现测试出来的 QPS=-1，QPS最小就是0了，不可能等于-1啊。另外不断报“float division by zero”的错。

## 使用的工具 / Tools Used
- [x] Perf / 模型推理压测工具

## 执行的代码或指令 / Code or Commands Executed

请提供您执行的主要代码或指令。 / Please provide the main code or commands you executed. 例如 / For example:

```bash
evalscope perf \
    --url "http://127.0.0.1:1234/v1/chat/completions" \
    --parallel 1 \
    --model qwen2.5 \
    --number 15 \
    --api openai \
    --dataset openqa \
    --stream
```

## 错误日志 / Error Log

请粘贴完整的错误日志或控制台输出。 / Please paste the full error log or console output. 例如 / For example:

```
evalscope perf \
    --url "http://127.0.0.1:1234/v1/chat/completions" \
    --parallel 1 \
    --model qwen2.5 \
    --number 15 \
    --api openai \
    --dataset openqa \
    --stream
2024-11-28 16:17:03,095 - datasets - INFO - PyTorch version 2.5.1 available.
2024-11-28 16:17:03,288 - evalscope - INFO - Starting benchmark...
2024-11-28 16:17:03,288 - evalscope - INFO - {
    "model": "qwen2.5",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": null,
    "url": "http://127.0.0.1:1234/v1/chat/completions",
    "headers": {
        "Authorization": "Bearer EMPTY"
    },
    "connect_timeout": 120,
    "read_timeout": 120,
    "api_key": "EMPTY",
    "number": 15,
    "parallel": 1,
    "rate": -1,
    "log_every_n_query": 10,
    "debug": false,
    "wandb_api_key": null,
    "name": null,
    "max_prompt_length": 9223372036854775807,
    "min_prompt_length": 0,
    "prompt": null,
    "query_template": null,
    "dataset": "openqa",
    "dataset_path": null,
    "frequency_penalty": null,
    "logprobs": null,
    "max_tokens": 2048,
    "min_tokens": null,
    "n_choices": null,
    "seed": 42,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": null,
    "top_p": null
}
2024-11-28 16:17:04,350 - evalscope - INFO - Connection successful.
2024-11-28 16:17:06,788 - evalscope - INFO - Save the result to: ./outputs/qwen2.5_perf/20241128_161706/benchmark_data.db
Processing: 0it [00:00, ?it/s]2024-11-28 16:17:25,527 - evalscope - WARNING - No usage info get.
2024-11-28 16:17:25,528 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/Users/ichen/.pyenv/versions/3.12.2/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 112, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
                              ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
Processing: 1it [00:18, 18.73s/it]2024-11-28 16:17:48,797 - evalscope - WARNING - No usage info get.
2024-11-28 16:17:48,797 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/Users/ichen/.pyenv/versions/3.12.2/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 112, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
                              ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
Processing: 2it [00:42, 21.40s/it]2024-11-28 16:18:14,901 - evalscope - WARNING - No usage info get.
2024-11-28 16:18:14,902 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/Users/ichen/.pyenv/versions/3.12.2/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 112, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
                              ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
Processing: 3it [01:08, 23.55s/it]2024-11-28 16:18:18,215 - evalscope - WARNING - No usage info get.
2024-11-28 16:18:18,215 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/Users/ichen/.pyenv/versions/3.12.2/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 112, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
                              ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
Processing: 4it [01:11, 15.56s/it]2024-11-28 16:18:31,288 - evalscope - WARNING - No usage info get.
2024-11-28 16:18:31,288 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/Users/ichen/.pyenv/versions/3.12.2/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 112, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
                              ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
Processing: 5it [01:24, 14.66s/it]2024-11-28 16:18:46,677 - evalscope - WARNING - No usage info get.
2024-11-28 16:18:46,677 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/Users/ichen/.pyenv/versions/3.12.2/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 112, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
                              ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
Processing: 6it [01:39, 14.91s/it]2024-11-28 16:19:04,704 - evalscope - WARNING - No usage info get.
2024-11-28 16:19:04,705 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/Users/ichen/.pyenv/versions/3.12.2/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 112, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
                              ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
Processing: 7it [01:57, 15.93s/it]2024-11-28 16:19:17,226 - evalscope - WARNING - No usage info get.
2024-11-28 16:19:17,226 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/Users/ichen/.pyenv/versions/3.12.2/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 112, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
                              ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
Processing: 8it [02:10, 14.84s/it]2024-11-28 16:19:37,710 - evalscope - WARNING - No usage info get.
2024-11-28 16:19:37,710 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/Users/ichen/.pyenv/versions/3.12.2/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 112, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
                              ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
Processing: 9it [02:30, 16.61s/it]2024-11-28 16:19:57,944 - evalscope - WARNING - No usage info get.
2024-11-28 16:19:57,944 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/Users/ichen/.pyenv/versions/3.12.2/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 112, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
                              ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
2024-11-28 16:19:57,948 - evalscope - INFO - {
  "Time taken for tests (senconds)": 171.142,
  "Number of concurrency": 1,
  "Total requests": 10,
  "Succeed requests": 10,
  "Failed requests": 0,
  "Average QPS": -1,
  "Average latency (s)": 17.114,
  "Average time to first token (s)": 0.355,
  "Average time per output token (s)": -1,
  "Average package latency (s)": 0.063,
  "Average package per request": 266.1,
  "Throughput(average output tokens per second)": 0.0,
  "Average input tokens per request": 0.0,
  "Average output tokens per request": 0.0
}
Processing: 10it [02:51, 17.73s/it]2024-11-28 16:20:06,528 - evalscope - WARNING - No usage info get.
2024-11-28 16:20:06,528 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/Users/ichen/.pyenv/versions/3.12.2/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 112, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
                              ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
Processing: 11it [02:59, 14.93s/it]2024-11-28 16:20:12,170 - evalscope - WARNING - No usage info get.
2024-11-28 16:20:12,171 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/Users/ichen/.pyenv/versions/3.12.2/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 112, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
                              ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
Processing: 12it [03:05, 12.10s/it]2024-11-28 16:20:30,360 - evalscope - WARNING - No usage info get.
2024-11-28 16:20:30,360 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/Users/ichen/.pyenv/versions/3.12.2/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 112, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
                              ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
Processing: 13it [03:23, 13.95s/it]2024-11-28 16:20:49,781 - evalscope - WARNING - No usage info get.
2024-11-28 16:20:49,781 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/Users/ichen/.pyenv/versions/3.12.2/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 112, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
                              ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
Processing: 14it [03:42, 15.60s/it]2024-11-28 16:21:08,937 - evalscope - WARNING - No usage info get.
2024-11-28 16:21:08,937 - evalscope - ERROR - float division by zero
Traceback (most recent call last):
  File "/Users/ichen/.pyenv/versions/3.12.2/lib/python3.12/site-packages/evalscope/perf/utils/benchmark_util.py", line 112, in calculate_averages
    self.avg_time_per_token = self.total_time / self.n_total_completion_tokens
                              ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: float division by zero
Processing: 15it [04:03, 16.21s/it]
2024-11-28 16:21:09,950 - evalscope - INFO -
Benchmarking summary:
+----------------------------------------------+----------------------------------------------------------+
| Key                                          | Value                                                    |
+==============================================+==========================================================+
| Time taken for tests (senconds)              | 242.134                                                  |
+----------------------------------------------+----------------------------------------------------------+
| Number of concurrency                        | 1                                                        |
+----------------------------------------------+----------------------------------------------------------+
| Total requests                               | 15                                                       |
+----------------------------------------------+----------------------------------------------------------+
| Succeed requests                             | 15                                                       |
+----------------------------------------------+----------------------------------------------------------+
| Failed requests                              | 0                                                        |
+----------------------------------------------+----------------------------------------------------------+
| Average QPS                                  | -1                                                       |
+----------------------------------------------+----------------------------------------------------------+
| Average latency (s)                          | 16.142                                                   |
+----------------------------------------------+----------------------------------------------------------+
| Average time to first token (s)              | 0.347                                                    |
+----------------------------------------------+----------------------------------------------------------+
| Average time per output token (s)            | -1                                                       |
+----------------------------------------------+----------------------------------------------------------+
| Average package latency (s)                  | 0.063                                                    |
+----------------------------------------------+----------------------------------------------------------+
| Average package per request                  | 251.6                                                    |
+----------------------------------------------+----------------------------------------------------------+
| Throughput(average output tokens per second) | 0.0                                                      |
+----------------------------------------------+----------------------------------------------------------+
| Average input tokens per request             | 0.0                                                      |
+----------------------------------------------+----------------------------------------------------------+
| Average output tokens per request            | 0.0                                                      |
+----------------------------------------------+----------------------------------------------------------+
| Expected number of requests                  | 15                                                       |
+----------------------------------------------+----------------------------------------------------------+
| Result DB path                               | ./outputs/qwen2.5_perf/20241128_161706/benchmark_data.db |
+----------------------------------------------+----------------------------------------------------------+
2024-11-28 16:21:09,956 - evalscope - INFO -
Percentile results:
+------------+-------------------------+-------------+
| Percentile | First Chunk Latency (s) | Latency (s) |
+------------+-------------------------+-------------+
|    10%     |         0.2838          |   5.6437    |
|    25%     |         0.2859          |   12.5216   |
|    50%     |         0.2888          |   18.189    |
|    66%     |         0.2895          |   19.1543   |
|    75%     |         0.5061          |   20.2335   |
|    80%     |          0.507          |   20.4824   |
|    90%     |         0.5089          |   23.2684   |
|    95%     |         0.5186          |   26.1047   |
|    98%     |         0.5186          |   26.1047   |
|    99%     |         0.5186          |   26.1047   |
+------------+-------------------------+-------------+
```

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
  - [ ] Windows
  - [x] macOS
  - [ ] Ubuntu

- Python版本 / Python Version:
  - [x] 3.12

## 其他信息 / Additional Information

如果有其他相关信息，请在此处提供。 / If there is any other relevant information, please provide it here.


加上`--debug`参数看看，估计是request返回的结果没有usage部分。如果没有返回usage，需要自己传`--tokenizer-path` 指定分词器权重路径，用于计算输入和输出的token数量

@Yunnglin 多谢答复，加上 --tokenizer-path 就正常了。

建议“evalscope - WARNING - No usage info get.”这个错误信息加上使用 --tokenizer-path 的说明