Repository: modelscope/evalscope
Issue: 华为 mindie 后端下 tokenizer统计结果错误
Root cause: Configuration Error

## 问题描述 / Issue Description

在华为国产服务器使用自定义tokenizer-path流式压测兼容 openai api 后端（这里使用的是 mindie，对Qwen2-7B-Instruct 进行推理，目录一致）

压测结果与 token 相关的全部明显不对劲。使用非流式并取消tokenizer-path，从后端接收 token 数量是没问题的：

+----------------------------------------------+-------------------------------------------------------+
| Throughput(average output tokens per second) | 10909.967                                             |
+----------------------------------------------+-------------------------------------------------------+
| Average output tokens per request            | 84558.8                                               |
+----------------------------------------------+-------------------------------------------------------+

## 使用的工具 / Tools Used

- [ ] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend 
- [ ] RAGEval backend
- [x] Perf / 模型推理压测工具
- [ ] Arena  /竞技场模式


## 执行的代码或指令 / Code or Commands Executed

请提供您执行的主要代码或指令。 / Please provide the main code or commands you executed. 例如 / For example:

```python
# 原谅我这么用，要对源代码做一些修改和 debug，同时Arguments.from_args 方法对这些参数的初始化不完全一致
command_line_args = [
            "--url", "http://127.0.0.1:1025/v1/chat/completions",
            "--parallel", "1",
            "-n", "5",
            "--model", "qwen",
            "--log-every-n-query", "10",
            "--read-timeout", "120",
            "--max-prompt-length", "8192",
            "--api", "openai",
            "--query-template",
            '{"model": "%m", "messages": [{"role": "user","content": "%p"}], "stream": true,"skip_special_tokens": false,"stop": ["
"]}',
            "--dataset", "openqa",
            "--wandb-api-key", "646f20bc5c7d7f3da**********",
            "--tokenizer-path", "/mnt/data/model/Qwen/Qwen2-7B-Instruct/Qwen/Qwen2-7B-Instruct",
            "--max-tokens", "4096",
            "--stream"
        ]
        task_cfg = command_line_args_to_dict(command_line_args)
        run_perf_benchmark(task_cfg)
```

## 错误日志 / Error Log

请粘贴完整的错误日志或控制台输出。 / Please paste the full error log or console output. 例如 / For example:

```shell
import sys; print('Python %s on %s' % (sys.version, sys.platform))
/root/anaconda3/envs/evalscope/bin/python -X pycache_prefix=/root/.cache/JetBrains/PyCharm2024.2/cpython-cache /root/.cache/JetBrains/RemoteDev/dist/fec67249e22be_pycharm-professional-2024.2.4-aarch64/plugins/python-ce/helpers/pydev/pydevd.py --multiprocess --qt-support=auto --client 127.0.0.1 --port 45971 --file /root/model/monitor_stress_testing.py 
已连接到 pydev 调试器(内部版本号 242.23726.102)2024-11-27 19:24:51,118 - datasets - INFO - PyTorch version 2.5.1 available.
压测Qwen2-7B-Instruct在1卡上的性能，使用并发1，请求100次:   0%|          | 0/61 [00:00<?, ?it/s]成功启动npu监控: {'message': 'Benchmark already running.'}
2024-11-27 19:24:52,378 - evalscope - INFO - Starting benchmark...
2024-11-27 19:24:52,378 - evalscope - INFO - {
    "model": "qwen",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": "/mnt/data/model/Qwen/Qwen2-7B-Instruct/Qwen/Qwen2-7B-Instruct",
    "url": "http://127.0.0.1:1025/v1/chat/completions",
    "headers": {
        "Authorization": "Bearer EMPTY"
    },
    "connect_timeout": 120,
    "read_timeout": 120,
    "api_key": "EMPTY",
    "number": 5,
    "parallel": 1,
    "rate": -1,
    "log_every_n_query": 10,
    "debug": false,
    "wandb_api_key": "646f20bc5c7d7f3da1b54a********",
    "name": null,
    "max_prompt_length": 8192,
    "min_prompt_length": 0,
    "prompt": null,
    "query_template": "{\"model\": \"%m\", \"messages\": [{\"role\": \"user\",\"content\": \"%p\"}], \"stream\": true,\"skip_special_tokens\": false,\"stop\": [\"
\"]}",
    "dataset": "openqa",
    "dataset_path": null,
    "frequency_penalty": null,
    "logprobs": null,
    "max_tokens": 4096,
    "min_tokens": null,
    "n_choices": null,
    "seed": 42,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": null,
    "top_p": null
}
2024-11-27 19:24:52,864 - evalscope - INFO - Connection successful.
2024-11-27 19:25:02,302 - evalscope - INFO - Save the result to: ./outputs/qwen_perf/20241127_192502/benchmark_data.db
Processing: 0it [00:00, ?it/s]
Processing: 1it [00:08,  8.29s/it]Token indices sequence length is longer than the specified maximum sequence length for this model (139645 > 131072). Running this sequence through the model will result in indexing errors
Processing: 2it [00:19,  9.97s/it]
Processing: 3it [00:30, 10.27s/it]
Processing: 4it [00:32,  7.31s/it]
Processing: 5it [00:39,  7.98s/it]
2024-11-27 19:25:51,724 - evalscope - INFO - 
Benchmarking summary:
+----------------------------------------------+-------------------------------------------------------+
| Key                                          | Value                                                 |
+==============================================+=======================================================+
| Time taken for tests (senconds)              | 38.753                                                |
+----------------------------------------------+-------------------------------------------------------+
| Number of concurrency                        | 1                                                     |
+----------------------------------------------+-------------------------------------------------------+
| Total requests                               | 5                                                     |
+----------------------------------------------+-------------------------------------------------------+
| Succeed requests                             | 5                                                     |
+----------------------------------------------+-------------------------------------------------------+
| Failed requests                              | 0                                                     |
+----------------------------------------------+-------------------------------------------------------+
| Average QPS                                  | 0.129                                                 |
+----------------------------------------------+-------------------------------------------------------+
| Average latency (s)                          | 7.444                                                 |
+----------------------------------------------+-------------------------------------------------------+
| Average time to first token (s)              | 0.085                                                 |
+----------------------------------------------+-------------------------------------------------------+
| Average time per output token (s)            | 9e-05                                                 |
+----------------------------------------------+-------------------------------------------------------+
| Average package latency (s)                  | 0.019                                                 |
+----------------------------------------------+-------------------------------------------------------+
| Average package per request                  | 381.2                                                 |
+----------------------------------------------+-------------------------------------------------------+
| Throughput(average output tokens per second) | 10909.967                                             |
+----------------------------------------------+-------------------------------------------------------+
| Average input tokens per request             | 18.4                                                  |
+----------------------------------------------+-------------------------------------------------------+
| Average output tokens per request            | 84558.8                                               |
+----------------------------------------------+-------------------------------------------------------+
| Expected number of requests                  | 5                                                     |
+----------------------------------------------+-------------------------------------------------------+
| Result DB path                               | ./outputs/qwen_perf/20241127_192502/benchmark_data.db |
+----------------------------------------------+-------------------------------------------------------+
2024-11-27 19:25:51,728 - evalscope - INFO - Too little data to calculate quantiles!
成功关闭npu监控: Benchmark stopped and final plot created.
```

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
  - [ ] Windows
  - [ ] macOS
  - [ ] Ubuntu
  - [x] 华为闭源欧拉

- Python版本 / Python Version:
  - [ ] 3.11
  - [x] 3.10
  - [ ] 3.9


## 其他信息 / Additional Information

以下给出outputs/qwen_perf/20241127_192502/benchmark_args.json

```json
{
    "model": "qwen",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": "/mnt/data/model/Qwen/Qwen2-7B-Instruct/Qwen/Qwen2-7B-Instruct",
    "url": "http://127.0.0.1:1025/v1/chat/completions",
    "headers": {
        "Authorization": "Bearer EMPTY"
    },
    "connect_timeout": 120,
    "read_timeout": 120,
    "api_key": "EMPTY",
    "number": 5,
    "parallel": 1,
    "rate": -1,
    "log_every_n_query": 10,
    "debug": false,
    "wandb_api_key": "646f20bc5c************",
    "name": null,
    "max_prompt_length": 8192,
    "min_prompt_length": 0,
    "prompt": null,
    "query_template": "{\"model\": \"%m\", \"messages\": [{\"role\": \"user\",\"content\": \"%p\"}], \"stream\": true,\"skip_special_tokens\": false,\"stop\": [\"
\"]}",
    "dataset": "openqa",
    "dataset_path": "./data/open_qa.jsonl",
    "frequency_penalty": null,
    "logprobs": null,
    "max_tokens": 4096,
    "min_tokens": null,
    "n_choices": null,
    "seed": 42,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": null,
    "top_p": null,
    "time": "20241127_192508",
  # detail这是我魔改了 wandb 模块，从环境变量拿我定义的描述
    "detail": "20241127192452: 压测Qwen2-7B-Instruct在1卡上的性能，使用并发1，请求100次"
}
```

/root/model/outputs/qwen_perf/20241127_192502/benchmark_summary.json

```json
{
    "Time taken for tests (senconds)": 38.753,
    "Number of concurrency": 1,
    "Total requests": 5,
    "Succeed requests": 5,
    "Failed requests": 0,
    "Average QPS": 0.129,
    "Average latency (s)": 7.444,
    "Average time to first token (s)": 0.085,
    "Average time per output token (s)": 9e-05,
    "Average package latency (s)": 0.019,
    "Average package per request": 381.2,
    "Throughput(average output tokens per second)": 10909.967,
    "Average input tokens per request": 18.4,
    "Average output tokens per request": 84558.8,
    "Expected number of requests": 5,
    "Result DB path": "./outputs/qwen_perf/20241127_192502/benchmark_data.db"
}
```



使用的是今天 20241127 18 点从 github clone 并 pip install e .的最新版本，包含了所有最新的更新（前天克隆的版本有一样的问题）
这个版本似乎解决了我在另一个地方提到的[流式卡主的问题](https://github.com/modelscope/evalscope/issues/200#issuecomment-2503448978)。非流式一会测试

```shell
(evalscope) ➜  evalscope git:(main) # 查看操作系统版本
cat /etc/os-release
NAME="EulerOS"
VERSION="2.0 (SP8)"
ID="euleros"
ID_LIKE="rhel fedora centos"
VERSION_ID="2.0"
PRETTY_NAME="EulerOS 2.0 (SP8)"
ANSI_COLOR="0;31"

(evalscope) ➜  evalscope git:(main) ✗ # 查看完整的系统和内核信息
uname -a
Linux bms-szl 4.19.36-vhulk1907.1.0.h1438.eulerosv2r8.aarch64 #1 SMP Sat May 6 17:11:15 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
(evalscope) ➜  evalscope git:(main) ✗ # 查看 CPU 信息
lscpu
Architecture:        aarch64
Byte Order:          Little Endian
CPU(s):              192
On-line CPU(s) list: 0-191
Thread(s) per core:  1
Core(s) per socket:  48
Socket(s):           4
NUMA node(s):        8
Vendor ID:           0x48
Model:               0
Stepping:            0x1
BogoMIPS:            200.00
L1d cache:           64K
L1i cache:           64K
L2 cache:            512K
L3 cache:            24576K
NUMA node0 CPU(s):   0-23
NUMA node1 CPU(s):   24-47
NUMA node2 CPU(s):   48-71
NUMA node3 CPU(s):   72-95
NUMA node4 CPU(s):   96-119
NUMA node5 CPU(s):   120-143
NUMA node6 CPU(s):   144-167
NUMA node7 CPU(s):   168-191
Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm```

显卡：华为 910A 32G X8

使用`--debug`参数看一下接受到的请求是什么样的，会输出类似下面格式：
```
2024-11-27 11:25:34,161 - evalscope - http_client.py - on_request_start - 116 - DEBUG - Starting request: <TraceRequestStartParams(method='POST', url=URL('http://127.0.0.1:8000/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench', 'Authorization': 'Bearer EMPTY')>)>
2024-11-27 11:25:34,163 - evalscope - http_client.py - on_request_chunk_sent - 128 - DEBUG - Request sent: <method='POST',  url=URL('http://127.0.0.1:8000/v1/completions'), truncated_chunk='{"prompt": "hello", "model": "qwen2.5"}'>
2024-11-27 11:25:38,172 - evalscope - http_client.py - on_response_chunk_received - 140 - DEBUG - Request received: <method='POST',  url=URL('http://127.0.0.1:8000/v1/completions'), truncated_chunk='{"id":"cmpl-a4565eb4fc6b4a5697f38c0adaf9b70b","object":"text_completion","created":1732677934,"model":"qwen2.5","choices":[{"index":0,"text":"，everyone！今天我给您撒个谎哦。 ))\\n\\n今天开心的事。","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":1,"total_tokens":17,"completion_tokens":16}}'>
```

启用debug 后的日志见文件。内容太多，编译器截断了一部分：
[log.md](https://github.com/user-attachments/files/17933837/log.md)


> 使用`--debug`参数看一下接受到的请求是什么样的，会输出类似下面格式：
> 
> ```
> 2024-11-27 11:25:34,161 - evalscope - http_client.py - on_request_start - 116 - DEBUG - Starting request: <TraceRequestStartParams(method='POST', url=URL('http://127.0.0.1:8000/v1/completions'), headers=<CIMultiDict('Content-Type': 'application/json', 'user-agent': 'modelscope_bench', 'Authorization': 'Bearer EMPTY')>)>
> 2024-11-27 11:25:34,163 - evalscope - http_client.py - on_request_chunk_sent - 128 - DEBUG - Request sent: <method='POST',  url=URL('http://127.0.0.1:8000/v1/completions'), truncated_chunk='{"prompt": "hello", "model": "qwen2.5"}'>
> 2024-11-27 11:25:38,172 - evalscope - http_client.py - on_response_chunk_received - 140 - DEBUG - Request received: <method='POST',  url=URL('http://127.0.0.1:8000/v1/completions'), truncated_chunk='{"id":"cmpl-a4565eb4fc6b4a5697f38c0adaf9b70b","object":"text_completion","created":1732677934,"model":"qwen2.5","choices":[{"index":0,"text":"，everyone！今天我给您撒个谎哦。 ))\\n\\n今天开心的事。","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":1,"total_tokens":17,"completion_tokens":16}}'>
> ```

您看看刚刚上传的文件。流式没有返回用量，因此我才启用的--tokenizer-path

如果取消流式，此时 token 数量因为是从 api 获取，是正确的，但是却没办法看到 first token 延迟

看了一下，这个流式返回的结果跟[openai api格式](https://platform.openai.com/docs/api-reference/chat/streaming)不太一样：

你的服务返回的是：
```
2024-11-27 19:48:52,142 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"id":"endpoint_common_9","object":"chat.completion.chunk","created":1732708132,"model":"qwen","choices":[{"index":0,"delta":{"role":"assistant","content":"PlayStation 2（PS2）和PlayStation Portable（PSP）是两个不同的游戏平台，它们的设计目的、硬件配置以及发布时间点都不同，因此在画面表现上也存在差异。\n\n1. **PS2的画面**：\n   - PS2发布于2000年，当时是次世代主机之一。它的图形处理能力在当时是非常先进的，能够提供高质量的3D游戏体验。\n   - PS2使用的是基于RISC架构的PowerPC处理器，并且配备了用于图形渲染的GPU单元。这使得它能够在2D和3D游戏中提供丰富的视觉效果，包括高分辨率纹理、动态光照和复杂的动画。\n   - 在游戏库中，PS2拥有大量的经典游戏，这些游戏在当时的画面标准下被认为是顶尖的。\n\n2. **PSP的画面**：\n   - PSP发布于2005年，是一款掌上游戏机。它的设计旨在提供便携性，同时保持相对较高的游戏性能。\n   - PSP主要依靠一个名为“Siggraph”的图形处理器，提供了比一些其他手持设备更高的分辨率和图形处理能力。它的屏幕尺寸和像素密度相比于PS2来说较小，但仍然可以提供良好的游戏体验。\n   - PSP支持的一些游戏，如《最终幻想》系列和《"},"finish_reason":null}]}
2024-11-27 19:48:52,160 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"id":"endpoint_common_9","object":"chat.completion.chunk","created":1732708132,"model":"qwen","choices":[{"index":0,"delta":{"role":"assistant","content":"PlayStation 2（PS2）和PlayStation Portable（PSP）是两个不同的游戏平台，它们的设计目的、硬件配置以及发布时间点都不同，因此在画面表现上也存在差异。\n\n1. **PS2的画面**：\n   - PS2发布于2000年，当时是次世代主机之一。它的图形处理能力在当时是非常先进的，能够提供高质量的3D游戏体验。\n   - PS2使用的是基于RISC架构的PowerPC处理器，并且配备了用于图形渲染的GPU单元。这使得它能够在2D和3D游戏中提供丰富的视觉效果，包括高分辨率纹理、动态光照和复杂的动画。\n   - 在游戏库中，PS2拥有大量的经典游戏，这些游戏在当时的画面标准下被认为是顶尖的。\n\n2. **PSP的画面**：\n   - PSP发布于2005年，是一款掌上游戏机。它的设计旨在提供便携性，同时保持相对较高的游戏性能。\n   - PSP主要依靠一个名为“Siggraph”的图形处理器，提供了比一些其他手持设备更高的分辨率和图形处理能力。它的屏幕尺寸和像素密度相比于PS2来说较小，但仍然可以提供良好的游戏体验。\n   - PSP支持的一些游戏，如《最终幻想》系列和《合金"},"finish_reason":null}]}
2024-11-27 19:48:52,179 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"id":"endpoint_common_9","object":"chat.completion.chunk","created":1732708132,"model":"qwen","choices":[{"index":0,"delta":{"role":"assistant","content":"PlayStation 2（PS2）和PlayStation Portable（PSP）是两个不同的游戏平台，它们的设计目的、硬件配置以及发布时间点都不同，因此在画面表现上也存在差异。\n\n1. **PS2的画面**：\n   - PS2发布于2000年，当时是次世代主机之一。它的图形处理能力在当时是非常先进的，能够提供高质量的3D游戏体验。\n   - PS2使用的是基于RISC架构的PowerPC处理器，并且配备了用于图形渲染的GPU单元。这使得它能够在2D和3D游戏中提供丰富的视觉效果，包括高分辨率纹理、动态光照和复杂的动画。\n   - 在游戏库中，PS2拥有大量的经典游戏，这些游戏在当时的画面标准下被认为是顶尖的。\n\n2. **PSP的画面**：\n   - PSP发布于2005年，是一款掌上游戏机。它的设计旨在提供便携性，同时保持相对较高的游戏性能。\n   - PSP主要依靠一个名为“Siggraph”的图形处理器，提供了比一些其他手持设备更高的分辨率和图形处理能力。它的屏幕尺寸和像素密度相比于PS2来说较小，但仍然可以提供良好的游戏体验。\n   - PSP支持的一些游戏，如《最终幻想》系列和《合金装备"},"finish_reason":null}]}
2024-11-27 19:48:52,197 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"id":"endpoint_common_9","object":"chat.completion.chunk","created":1732708132,"model":"qwen","choices":[{"index":0,"delta":{"role":"assistant","content":"PlayStation 2（PS2）和PlayStation Portable（PSP）是两个不同的游戏平台，它们的设计目的、硬件配置以及发布时间点都不同，因此在画面表现上也存在差异。\n\n1. **PS2的画面**：\n   - PS2发布于2000年，当时是次世代主机之一。它的图形处理能力在当时是非常先进的，能够提供高质量的3D游戏体验。\n   - PS2使用的是基于RISC架构的PowerPC处理器，并且配备了用于图形渲染的GPU单元。这使得它能够在2D和3D游戏中提供丰富的视觉效果，包括高分辨率纹理、动态光照和复杂的动画。\n   - 在游戏库中，PS2拥有大量的经典游戏，这些游戏在当时的画面标准下被认为是顶尖的。\n\n2. **PSP的画面**：\n   - PSP发布于2005年，是一款掌上游戏机。它的设计旨在提供便携性，同时保持相对较高的游戏性能。\n   - PSP主要依靠一个名为“Siggraph”的图形处理器，提供了比一些其他手持设备更高的分辨率和图形处理能力。它的屏幕尺寸和像素密度相比于PS2来说较小，但仍然可以提供良好的游戏体验。\n   - PSP支持的一些游戏，如《最终幻想》系列和《合金装备》"},"finish_reason":null}]}
2024-11-27 19:48:52,216 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"id":"endpoint_common_9","object":"chat.completion.chunk","created":1732708132,"model":"qwen","choices":[{"index":0,"delta":{"role":"assistant","content":"PlayStation 2（PS2）和PlayStation Portable（PSP）是两个不同的游戏平台，它们的设计目的、硬件配置以及发布时间点都不同，因此在画面表现上也存在差异。\n\n1. **PS2的画面**：\n   - PS2发布于2000年，当时是次世代主机之一。它的图形处理能力在当时是非常先进的，能够提供高质量的3D游戏体验。\n   - PS2使用的是基于RISC架构的PowerPC处理器，并且配备了用于图形渲染的GPU单元。这使得它能够在2D和3D游戏中提供丰富的视觉效果，包括高分辨率纹理、动态光照和复杂的动画。\n   - 在游戏库中，PS2拥有大量的经典游戏，这些游戏在当时的画面标准下被认为是顶尖的。\n\n2. **PSP的画面**：\n   - PSP发布于2005年，是一款掌上游戏机。它的设计旨在提供便携性，同时保持相对较高的游戏性能。\n   - PSP主要依靠一个名为“Siggraph”的图形处理器，提供了比一些其他手持设备更高的分辨率和图形处理能力。它的屏幕尺寸和像素密度相比于PS2来说较小，但仍然可以提供良好的游戏体验。\n   - PSP支持的一些游戏，如《最终幻想》系列和《合金装备》系列"},"finish_reason":null}]}
2024-11-27 19:48:52,234 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"id":"endpoint_common_9","object":"chat.completion.chunk","created":1732708132,"model":"qwen","choices":[{"index":0,"delta":{"role":"assistant","content":"PlayStation 2（PS2）和PlayStation Portable（PSP）是两个不同的游戏平台，它们的设计目的、硬件配置以及发布时间点都不同，因此在画面表现上也存在差异。\n\n1. **PS2的画面**：\n   - PS2发布于2000年，当时是次世代主机之一。它的图形处理能力在当时是非常先进的，能够提供高质量的3D游戏体验。\n   - PS2使用的是基于RISC架构的PowerPC处理器，并且配备了用于图形渲染的GPU单元。这使得它能够在2D和3D游戏中提供丰富的视觉效果，包括高分辨率纹理、动态光照和复杂的动画。\n   - 在游戏库中，PS2拥有大量的经典游戏，这些游戏在当时的画面标准下被认为是顶尖的。\n\n2. **PSP的画面**：\n   - PSP发布于2005年，是一款掌上游戏机。它的设计旨在提供便携性，同时保持相对较高的游戏性能。\n   - PSP主要依靠一个名为“Siggraph”的图形处理器，提供了比一些其他手持设备更高的分辨率和图形处理能力。它的屏幕尺寸和像素密度相比于PS2来说较小，但仍然可以提供良好的游戏体验。\n   - PSP支持的一些游戏，如《最终幻想》系列和《合金装备》系列，"},"finish_reason":null}]}
```

正常格式是：
```
2024-11-27 20:02:24,406 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"。同时"},"finish_reason":null}],"usage":null}
2024-11-27 20:02:24,449 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":""},"finish_reason":null}],"usage":null}
2024-11-27 20:02:24,492 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"，保持"},"finish_reason":null}],"usage":null}
2024-11-27 20:02:24,536 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"与"},"finish_reason":null}],"usage":null}
2024-11-27 20:02:24,579 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"运营商"},"finish_reason":null}],"usage":null}
2024-11-27 20:02:24,622 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"的良好"},"finish_reason":null}],"usage":null}
2024-11-27 20:02:24,676 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"沟通"},"finish_reason":null}],"usage":null}
2024-11-27 20:02:24,720 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"是非常"},"finish_reason":null}],"usage":null}
2024-11-27 20:02:24,760 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"重要的"},"finish_reason":null}],"usage":null}
2024-11-27 20:02:24,803 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":""},"finish_reason":null}],"usage":null}
2024-11-27 20:02:24,847 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"，以便"},"finish_reason":null}],"usage":null}
2024-11-27 20:02:24,890 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"及时"},"finish_reason":null}],"usage":null}
2024-11-27 20:02:24,933 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"得到"},"finish_reason":null}],"usage":null}
2024-11-27 20:02:24,976 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"帮助"},"finish_reason":null}],"usage":null}
2024-11-27 20:02:25,023 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"和支持"},"finish_reason":null}],"usage":null}
2024-11-27 20:02:25,066 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":""},"finish_reason":null}],"usage":null}
2024-11-27 20:02:25,109 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":""},"finish_reason":null}],"usage":null}
2024-11-27 20:02:25,111 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"。
"},"finish_reason":null}],"usage":null}
2024-11-27 20:02:25,113 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: {"model":"Qwen2.5-0.5B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":50,"completion_tokens":260,"total_tokens":310}}
2024-11-27 20:02:25,113 - evalscope - http_client.py - _handle_stream - 57 - DEBUG - Response recevied: data: [DONE]
```

返回的token太多，导致评估结果不对

这么说确实是，正常是返回一小部分读取的，也不知道华为 mindie 为什么这么干
有没有曲线救国方法？感觉可以修改代码读取最后完整输出的时候，再计算 token

需要自己修改`parse_responses()`方法来计算token数量了，需要计算两个chunk之间的token差值

参考：[自定义API](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/custom.html#api)

收到，感谢

> 需要自己修改`parse_responses()`方法来计算token数量了，需要计算两个chunk之间的token差值
> 
> 参考：[自定义API](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/custom.html#api)



如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！

补充一下，你这边其实只用计算最后一个chunk的token数量即可，因为最后一个是完整的，中间的token数不用记录

> 补充一下，你这边其实只用计算最后一个chunk的token数量即可，因为最后一个是完整的，中间的token数不用记录

感谢提醒，这个清楚的

> 如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！

不需要提醒，早早点上了😀

给使用mindie server的兄弟提个醒，这个问题出在一个配置上：
`"fullTextEnabled": false,`
指定为false或者不填默认是false，关闭流式接口全量返回历史结果，此时即可正常返回。
```shell
+----------------------------------------------+-------------------------------------------------------+
| Key                                          | Value                                                 |
+==============================================+=======================================================+
| Time taken for tests (senconds)              | 36.623                                                |
+----------------------------------------------+-------------------------------------------------------+
| Number of concurrency                        | 1                                                     |
+----------------------------------------------+-------------------------------------------------------+
| Total requests                               | 5                                                     |
+----------------------------------------------+-------------------------------------------------------+
| Succeed requests                             | 5                                                     |
+----------------------------------------------+-------------------------------------------------------+
| Failed requests                              | 0                                                     |
+----------------------------------------------+-------------------------------------------------------+
| Average QPS                                  | 0.137                                                 |
+----------------------------------------------+-------------------------------------------------------+
| Average latency (s)                          | 7.305                                                 |
+----------------------------------------------+-------------------------------------------------------+
| Average time to first token (s)              | 0.1                                                   |
+----------------------------------------------+-------------------------------------------------------+
| Average time per output token (s)            | 0.01916                                               |
+----------------------------------------------+-------------------------------------------------------+
| Average package latency (s)                  | 0.019                                                 |
+----------------------------------------------+-------------------------------------------------------+
| Average package per request                  | 381.2                                                 |
+----------------------------------------------+-------------------------------------------------------+
| Throughput(average output tokens per second) | 52.181                                                |
+----------------------------------------------+-------------------------------------------------------+
| Average input tokens per request             | 18.4                                                  |
+----------------------------------------------+-------------------------------------------------------+
| Average output tokens per request            | 382.2                                                 |
+----------------------------------------------+-------------------------------------------------------+
| Expected number of requests                  | 5                                                     |
+----------------------------------------------+-------------------------------------------------------+
| Result DB path                               | ./outputs/qwen_perf/20241127_205118/benchmark_data.db |
+----------------------------------------------+-------------------------------------------------------+
```