Repository: modelscope/evalscope
Issue: - `--min-tokens` 生成的最少token数量，不是所有模型服务都支持该参数，请查看对应API文档。
Root cause: External Dependency Breakage

- `--min-tokens` 生成的最少token数量，不是所有模型服务都支持该参数，请查看对应API文档。
请问是查 谁的 对应的API文档呢？ 模型的？ 推理引擎的？

你测试的模型服务的文档，就是谁提供的API服务，可能是推理引擎拉起的服务，也可能是云服务商提供的服务

> 你测试的模型服务的文档，就是谁提供的API服务，可能是推理引擎拉起的服务，也可能是云服务商提供的服务

谢谢回复。我用vllm serve的qwen 32b instruct，然后

from evalscope.perf.main import run_perf_benchmark
task_cfg = {"url": "http://127.0.0.1:9000/v1/chat/completions",
            "parallel": 1,
            "model": "Qwen/Qwen2.5-32B-Instruct",
            "number": 15,
            "api": "openai",
            "dataset": "random",
            "prefix_length": 0,
            "min_prompt_length": 1024,
            "max_prompt_length": 1024,
            "max_tokens":1024,
            "min_tokens":1024,
            "tokenizer_path": "Qwen/Qwen2.5-32B-Instruct",
            "stream": True}
但是测试结果显示output tokens都不到1024
+------------+----------+----------+-------------+--------------+---------------+----------------------+
| Percentile | TTFT (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Throughput(tokens/s) |
+------------+----------+----------+-------------+--------------+---------------+----------------------+
|    10%     |  0.0345  |  0.0274  |   1.1371    |     1024     |      41       |       35.7052        |
|    25%     |  0.0382  |  0.0277  |   1.8121    |     1031     |      65       |       35.8179        |
|    50%     |  0.0403  |  0.0277  |   2.6766    |     1068     |      96       |       35.8663        |
|    66%     |  0.0408  |  0.0278  |   2.9732    |     1071     |      107      |       35.8922        |
|    75%     |  0.042   |  0.0278  |   3.2268    |     1090     |      116      |        35.976        |
|    80%     |  0.042   |  0.0278  |   3.3184    |     1208     |      119      |       35.9883        |
|    90%     |  0.042   |  0.028   |    3.638    |     1211     |      131      |       36.0086        |
|    95%     |  0.0431  |  0.0285  |   4.2668    |     1324     |      153      |       36.0556        |
|    98%     |  0.0431  |  0.0299  |   4.2668    |     1324     |      153      |       36.0556        |
|    99%     |  0.0431  |  0.0317  |   4.2668    |     1324     |      153      |       36.0556        |
+------------+----------+----------+-------------+--------------+---------------+----------------------+

使用的evalscope是main分支代码吗，用的vllm是什么版本呢，我使用 vllm v0.7.2 没有问题

```python
from evalscope.perf.arguments import Arguments
task_cfg = Arguments(
    parallel=20,
    model='Qwen2.5-0.5B-Instruct',
    url='http://127.0.0.1:8801/v1/chat/completions',
    api='openai',
    dataset='random',
    min_tokens=1024,
    max_tokens=1024,
    prefix_length=0,
    min_prompt_length=1024,
    max_prompt_length=1024,
    number=40,
    tokenizer_path='Qwen/Qwen2.5-0.5B-Instruct',
    seed=None,
    debug= True,
)
run_perf_benchmark(task_cfg)
```


```text
+------------+----------+----------+-------------+--------------+---------------+----------------------+
| Percentile | TTFT (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Throughput(tokens/s) |
+------------+----------+----------+-------------+--------------+---------------+----------------------+
|    10%     |  4.5325  |   nan    |   4.5325    |     1024     |     1024      |       223.4209       |
|    25%     |  4.5331  |   nan    |   4.5331    |     1051     |     1024      |       223.618        |
|    50%     |  4.5739  |   nan    |   4.5739    |     1071     |     1024      |       225.8485       |
|    66%     |  4.5769  |   nan    |   4.5769    |     1080     |     1024      |       225.8823       |
|    75%     |  4.5799  |   nan    |   4.5799    |     1085     |     1024      |       225.8979       |
|    80%     |  4.5806  |   nan    |   4.5806    |     1087     |     1024      |       225.9121       |
|    90%     |  4.5847  |   nan    |   4.5847    |     1098     |     1024      |       225.9321       |
|    95%     |  4.5854  |   nan    |   4.5854    |     1270     |     1024      |       225.9383       |
|    98%     |  4.5876  |   nan    |   4.5876    |     1310     |     1024      |       225.9551       |
|    99%     |  4.5876  |   nan    |   4.5876    |     1310     |     1024      |       225.9551       |
+------------+----------+----------+-------------+--------------+---------------+----------------------+
```

谢谢回复。之前的环境：今天git clone 的evalscope code 然后编译，今天 pip install vllm （即0.8.1）；我把vllm换成了0.7.2，就ok了。
btw：如果是直接pip install evalscope，运行上述测试指令，会报错，不识别参数prefix_length。

+------------+----------+----------+-------------+--------------+---------------+----------------------+
| Percentile | TTFT (s) | TPOT (s) | Latency (s) | Input tokens | Output tokens | Throughput(tokens/s) |
+------------+----------+----------+-------------+--------------+---------------+----------------------+
|    10%     | 27.9186  |   nan    |   27.9186   |     1025     |     1024      |       36.4855        |
|    25%     | 27.9224  |   nan    |   27.9224   |     1034     |     1024      |       36.5532        |
|    50%     | 27.9457  |   nan    |   27.9457   |     1069     |     1024      |       36.6425        |
|    66%     | 27.9614  |   nan    |   27.9614   |     1072     |     1024      |       36.6544        |
|    75%     | 28.0139  |   nan    |   28.0139   |     1091     |     1024      |       36.6731        |
|    80%     | 28.0386  |   nan    |   28.0386   |     1209     |     1024      |       36.6731        |
|    90%     | 28.0659  |   nan    |   28.0659   |     1212     |     1024      |        36.678        |
|    95%     | 28.0739  |   nan    |   28.0739   |     1325     |     1024      |       36.6826        |
|    98%     | 28.0739  |   nan    |   28.0739   |     1325     |     1024      |       36.6826        |
|    99%     | 28.0739  |   nan    |   28.0739   |     1325     |     1024      |       36.6826        |
+------------+----------+----------+-------------+--------------+---------------+----------------------+






是的，pypi 包还没有支持这个random功能，更新稍落后

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！