Repository: modelscope/evalscope
Issue: stream参数的开启对平均延时的结果差异上很大
Root cause: Algorithmic Error

## 问题描述 / Issue Description

evalscope使用stream压测和关闭stream在平均时延时间上相差巨大

## 使用的工具 / Tools Used
- [ ] Perf / 模型推理压测工具


## 执行的代码或指令 / Code or Commands Executed
evalscope perf --model Qwen2.5-72B-Instruct --url http://192.168.100.30:8001/v1/chat/completions --parallel 64 --number 64 --api openai --dataset openqa
# 开启流式
evalscope perf --model Qwen2.5-72B-Instruct --url http://192.168.100.30:8001/v1/chat/completions --parallel 64 --number 64 --api openai --dataset openqa --stream

## 错误日志 / Error Log
evalscope版本：evalscope 0.11.0

开启流式的结果：
Benchmarking summary:
+-----------------------------------+----------------------------------------------------------------+
| Key                               | Value                                                          |
+===================================+================================================================+
| Time taken for tests (s)          | 452.747                                                        |
+-----------------------------------+----------------------------------------------------------------+
| Number of concurrency             | 64                                                             |
+-----------------------------------+----------------------------------------------------------------+
| Total requests                    | 64                                                             |
+-----------------------------------+----------------------------------------------------------------+
| Succeed requests                  | 63                                                             |
+-----------------------------------+----------------------------------------------------------------+
| Failed requests                   | 1                                                              |
+-----------------------------------+----------------------------------------------------------------+
| Throughput(average tokens/s)      | 40.672                                                         |
+-----------------------------------+----------------------------------------------------------------+
| Average QPS                       | 0.139                                                          |
+-----------------------------------+----------------------------------------------------------------+
| Average latency (s)               | 320.319                                                        |
+-----------------------------------+----------------------------------------------------------------+
| Average time to first token (s)   | 2.827                                                          |
+-----------------------------------+----------------------------------------------------------------+
| Average time per output token (s) | 0.02459                                                        |
+-----------------------------------+----------------------------------------------------------------+
| Average input tokens per request  | 50.73                                                          |
+-----------------------------------+----------------------------------------------------------------+
| Average output tokens per request | 292.286                                                        |
+-----------------------------------+----------------------------------------------------------------+
| Average package latency (s)       | 1.086                                                          |
+-----------------------------------+----------------------------------------------------------------+
| Average package per request       | 292.286                                                        |
+-----------------------------------+----------------------------------------------------------------+
| Expected number of requests       | 64                                                             |
+-----------------------------------+----------------------------------------------------------------+
| Result DB path                    | outputs/20250219_164728/Qwen2.5-72B-Instruct/benchmark_data.db |
+-----------------------------------+----------------------------------------------------------------+
2025-02-19 16:55:05,929 - evalscope - INFO - 
Percentile results:
+------------+----------+----------+-------------+--------------+---------------+----------------------+
| Percentile | TTFT (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Throughput(tokens/s) |
+------------+----------+----------+-------------+--------------+---------------+----------------------+
|    10%     |  1.4642  |  0.0197  |  150.9679   |      42      |      122      |        0.5468        |
|    25%     |  2.0914  |  0.0208  |  272.5447   |      45      |      189      |        0.6451        |
|    50%     |  2.8684  |  0.0228  |  344.2743   |      50      |      308      |        0.8598        |
|    66%     |  3.316   |  0.0247  |  379.3539   |      53      |      354      |        1.0762        |
|    75%     |  3.6493  |  0.0263  |  395.0565   |      55      |      398      |        1.2495        |
|    80%     |  3.7882  |  0.0275  |  404.4733   |      55      |      407      |        1.3267        |
|    90%     |  4.0824  |  0.031   |  422.6837   |      61      |      441      |        1.4601        |
|    95%     |  4.2166  |  0.0344  |  434.4725   |      64      |      456      |        1.5352        |
|    98%     |  4.2953  |  0.0392  |  446.9334   |      66      |      504      |        1.7006        |
|    99%     |  4.3461  |  0.0494  |  452.5605   |      66      |      597      |        3.5048        |
+------------+----------+----------+-------------+--------------+---------------+----------------------+


不开启流式的结果：
Benchmarking summary:
+-----------------------------------+----------------------------------------------------------------+
| Key                               | Value                                                          |
+===================================+================================================================+
| Time taken for tests (s)          | 70.838                                                         |
+-----------------------------------+----------------------------------------------------------------+
| Number of concurrency             | 64                                                             |
+-----------------------------------+----------------------------------------------------------------+
| Total requests                    | 64                                                             |
+-----------------------------------+----------------------------------------------------------------+
| Succeed requests                  | 64                                                             |
+-----------------------------------+----------------------------------------------------------------+
| Failed requests                   | 0                                                              |
+-----------------------------------+----------------------------------------------------------------+
| Throughput(average tokens/s)      | 271.352                                                        |
+-----------------------------------+----------------------------------------------------------------+
| Average QPS                       | 0.903                                                          |
+-----------------------------------+----------------------------------------------------------------+
| Average latency (s)               | 40.94                                                          |
+-----------------------------------+----------------------------------------------------------------+
| Average time to first token (s)   | 40.94                                                          |
+-----------------------------------+----------------------------------------------------------------+
| Average time per output token (s) | 0.00369                                                        |
+-----------------------------------+----------------------------------------------------------------+
| Average input tokens per request  | 50.594                                                         |
+-----------------------------------+----------------------------------------------------------------+
| Average output tokens per request | 300.344                                                        |
+-----------------------------------+----------------------------------------------------------------+
| Average package latency (s)       | 40.94                                                          |
+-----------------------------------+----------------------------------------------------------------+
| Average package per request       | 1.0                                                            |
+-----------------------------------+----------------------------------------------------------------+
| Expected number of requests       | 64                                                             |
+-----------------------------------+----------------------------------------------------------------+
| Result DB path                    | outputs/20250219_163022/Qwen2.5-72B-Instruct/benchmark_data.db |
+-----------------------------------+----------------------------------------------------------------+
2025-02-19 16:31:37,658 - evalscope - INFO - 
Percentile results:
+------------+----------+----------+-------------+--------------+---------------+----------------------+
| Percentile | TTFT (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Throughput(tokens/s) |
+------------+----------+----------+-------------+--------------+---------------+----------------------+
|    10%     | 20.3882  |   nan    |   20.3882   |      42      |      122      |        5.9838        |
|    25%     | 30.7918  |   nan    |   30.7918   |      45      |      190      |        6.1715        |
|    50%     | 44.4495  |   nan    |   44.4495   |      50      |      308      |        6.9292        |
|    66%     | 50.0306  |   nan    |   50.0306   |      53      |      375      |        7.4954        |
|    75%     | 52.4607  |   nan    |   52.4607   |      55      |      405      |        7.7238        |
|    80%     | 52.9897  |   nan    |   52.9897   |      55      |      414      |        7.8128        |
|    90%     | 54.6702  |   nan    |   54.6702   |      61      |      443      |        8.1031        |
|    95%     | 55.7312  |   nan    |   55.7312   |      64      |      463      |        8.3077        |
|    98%     |  62.576  |   nan    |   62.576    |      66      |      597      |        9.5404        |
|    99%     |  70.838  |   nan    |   70.838    |      66      |      763      |        10.771        |
+------------+----------+----------+-------------+--------------+---------------+----------------------+


从两次结果上来看，开启stream后平均延时很大，并且通过对比大模型本身的监控：

![Image](https://github.com/user-attachments/assets/04d2fc1d-cb05-4532-b9c9-526c30dad04b)
整个延时并没有那么多。

自行写了简单的python脚本加流式，并发64个，结果和不加流式的差不多。
请问这个是什么问题？
```

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
  - [ ] Ubuntu

- Python版本 / Python Version:
  - [ ] 3.10



我这边自己测试vllm部署的服务，加不加stream，结果也差不多诶，你再试一下呢

加stream
```json
{
    "Time taken for tests (s)": 26.739,
    "Number of concurrency": 64,
    "Total requests": 64,
    "Succeed requests": 64,
    "Failed requests": 0,
    "Throughput(average tokens/s)": 1913.246,
    "Average QPS": 2.393,
    "Average latency (s)": 13.797,
    "Average time to first token (s)": 0.248,
    "Average time per output token (s)": 0.00052,
    "Average input tokens per request": 26.594,
    "Average output tokens per request": 799.359,
    "Average package latency (s)": 0.017,
    "Average package per request": 799.359,
    "Expected number of requests": 64,
    "Result DB path": "outputs/20250219_172536/DeepSeek-R1-Distill-Qwen-7B/benchmark_data.db"
}
```

不加stream
```json
{
    "Time taken for tests (s)": 24.268,
    "Number of concurrency": 64,
    "Total requests": 64,
    "Succeed requests": 64,
    "Failed requests": 0,
    "Throughput(average tokens/s)": 2116.797,
    "Average QPS": 2.637,
    "Average latency (s)": 13.56,
    "Average time to first token (s)": 13.56,
    "Average time per output token (s)": 0.00047,
    "Average input tokens per request": 26.594,
    "Average output tokens per request": 802.672,
    "Average package latency (s)": 13.56,
    "Average package per request": 1.0,
    "Expected number of requests": 64,
    "Result DB path": "outputs/20250219_172413/DeepSeek-R1-Distill-Qwen-7B/benchmark_data.db"
}
```

我试过很多次了，在我这边时间就很大，之前还怀疑是vllm参数问题，今天才发现stream参数的影响，有什么办法可以排查这个问题吗，或者能在哪边把每个请求的具体时间输出出来

可以加上 --debug ，会输出具体的请求

--debug看不出来什么问题，日志一直在打印接收到的流

我看你提供的log，加上stream还出现了请求失败的情况，每个请求内部的token吞吐速度也比较慢。
```text
+-----------------------------------+----------------------------------------------------------------+
| Total requests | 64 |
+-----------------------------------+----------------------------------------------------------------+
| Succeed requests | 63 |
+-----------------------------------+----------------------------------------------------------------+
| Failed requests | 1 |
+-----------------------------------+----------------------------------------------------------------+
```
请问使用的是什么版本的vllm，换一个小一点的模型也是这样吗？

请问TPOP值为空是为什么怎么解决

> 请问TPOP值为空是为什么怎么解决

tpop需要指定stream模式才能计算

> > 请问TPOP值为空是为什么怎么解决
> 
> tpop需要指定stream模式才能计算

非常感谢您的回复，在别的问题中找到了答案，还是非常感谢您，还有请问我进行并发测试的时候，输入较短的时候可以使用openqa的数据集进行测试，如果我想测试长输入例如2000token以上，我出来单一设置prompt外，还有什么数据集或者其他方式可以像openqa这样使用简单。

> > > 请问TPOP值为空是为什么怎么解决
> > 
> > tpop需要指定stream模式才能计算
> 
> 非常感谢您的回复，在别的问题中找到了答案，还是非常感谢您，还有请问我进行并发测试的时候，输入较短的时候可以使用openqa的数据集进行测试，如果我想测试长输入例如2000token以上，我出来单一设置prompt外，还有什么数据集或者其他方式可以像openqa这样使用简单。

还支持longalpaca这个数据集，prompt会长一点，也支持自定义数据集 参考https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/parameters.html#id5

我上周五新建了一台Centos7.9的机器重新测试了下，平均延时正常了，之前用的是新装的Ubuntu24.04版本。
今天又新建了一台Ubuntu24.04机器，重新压测后，平均延时也没有问题了。
目前怀疑最早测试的那台Ubuntu24.04可能哪边有啥问题，但是目前还没排查到具体原因。工具本身是没问题的。

再请教下 --min-prompt-length 和 --max-prompt-length 参数，看文档是指输入的prompt的最小和最大长度，但是prompt是来自数据集里的吧，那设置这两个参数有什么意义？比如数据集里都是tokens在300的长度，这两个参数会产生什么样的效果呢

参考下面的代码，就是对prompt长度进行一个简单过滤，筛选特定范围的prompt：

https://github.com/modelscope/evalscope/blob/5bc798a74e8c153543149aa0099228fd78044e9c/evalscope/perf/plugin/datasets/openqa.py#L32-L37

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！