Repository: modelscope/evalscope
Issue:  Too little data to calculate quantiles!
Root cause: Validation Gap

## 问题描述 / Issue Description

请简要描述您遇到的问题。 / Please briefly describe the issue you encountered.

## 使用的工具 / Tools Used
- [ ] Perf / 模型推理压测工具


## 执行的代码或指令 / Code or Commands Executed

请提供您执行的主要代码或指令。 / Please provide the main code or commands you executed. 例如 / For example:

```python
from evalscope.perf.main import run_perf_benchmark

task_cfg = {"url": "http://localhost:52415/v1/chat/completions",
            "parallel": 4,
            "model": "qwen/Qwen1.5-7B-Chat",
            "number": 15,
            "api": "openai",
            "dataset": "openqa",
            "stream": True}
run_perf_benchmark(task_cfg)
```

```bash
python benchmark.py
```

## 错误日志 / Error Log

请粘贴完整的错误日志或控制台输出。 / Please paste the full error log or console output. 例如 / For example:

```
2024-12-18 15:59:48,057 - datasets - INFO - PyTorch version 2.5.1 available.
2024-12-18 15:59:48,307 - evalscope - INFO - Save the result to: ./outputs/20241218_155948/Qwen1.5-7B-Chat
2024-12-18 15:59:48,308 - evalscope - INFO - Starting benchmark...
2024-12-18 15:59:48,308 - evalscope - INFO - {
    "model": "qwen/Qwen1.5-7B-Chat",
    "model_id": "Qwen1.5-7B-Chat",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": null,
    "port": 8877,
    "url": "http://localhost:52415/v1/chat/completions",
    "headers": {
        "Authorization": "Bearer EMPTY"
    },
    "connect_timeout": 120,
    "read_timeout": 120,
    "api_key": "EMPTY",
    "number": 15,
    "parallel": 4,
    "rate": -1,
    "log_every_n_query": 10,
    "debug": false,
    "wandb_api_key": null,
    "name": null,
    "outputs_dir": "./outputs/20241218_155948/Qwen1.5-7B-Chat",
    "max_prompt_length": 9223372036854775807,
    "min_prompt_length": 0,
    "prompt": null,
    "query_template": null,
    "dataset": "openqa",
    "dataset_path": null,
    "frequency_penalty": null,
    "logprobs": null,
    "max_tokens": 2048,
    "min_tokens": null,
    "n_choices": null,
    "seed": 42,
    "stop": [],
    "stop_token_ids": [],
    "stream": true,
    "temperature": null,
    "top_p": null
}
2024-12-18 15:59:48,450 - evalscope - INFO - Connection successful.
2024-12-18 15:59:50,825 - evalscope - INFO - Save the data base to: ./outputs/20241218_155948/Qwen1.5-7B-Chat/benchmark_data.db
Processing: 0it [00:00, ?it/s]2024-12-18 15:59:50,831 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '盗贼天赋盗贼怎么加天赋?知道告诉一下下啦~~'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,832 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': 'Ps2比 画面好吗?差距很大吗? Ps2比psp画面好吗?差距很大吗?'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,832 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '加装方向助力我有一普桑想加装一套方向助力可以吗？？怎么改？'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,833 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '比斯巨兽的皮干什么用的？多少钱？'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,834 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '神州行手机号码成为空号了怎么办，卡里还有几十圆钱，还可以再用吗'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,834 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '最近脸上长了好多脂肪粒，这个该怎么破？ 记得前段时间你好像也有～'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,835 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '菠营养很丰富,那每天吃多少为益? 菠菜能多吃吗?菠菜营养很丰富,那每天吃多少为益?'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,836 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '哈尔滨院附近的旅店多吗？价位大概是什么样的 哈尔滨金融学院附近的旅店多吗？价位大概是什么样的'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,836 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '天津大悦城附近有什么好吃的？3个人吃'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,837 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '怎样向中的女孩子说出第一句话？说什么？ 怎样向暗恋中的女孩子说出第一句话？说什么？'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,838 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '我想买一辆童车，请问什么牌子的好？济南的专卖童车的地方在那儿？'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,838 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '小伙伴们 这个叫什么名字 求科普[玫瑰][玫瑰][玫瑰]'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,839 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '怎么把物品捐给和尚道士加声望，他们不理我'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,839 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '玩什么好合欢105级玩什么法宝好是PK的哦  合欢105级    玩什么法宝好    是PK的哦'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,840 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '请问编辑数码照片的那个比较好用？ 请问编辑数码照片的软件那个比较好用？'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,840 - evalscope - INFO - {
  "Time taken for tests (s)": 0.013,
  "Number of concurrency": 4,
  "Total requests": 10,
  "Succeed requests": 0,
  "Failed requests": 10,
  "Throughput(average tokens/s)": -1,
  "Average QPS": -1,
  "Average latency (s)": -1,
  "Average time to first token (s)": -1,
  "Average time per output token (s)": -1,
  "Average input tokens per request": -1,
  "Average output tokens per request": -1,
  "Average package latency (s)": -1,
  "Average package per request": -1
}
Processing: 15it [00:01, 14.75it/s]
2024-12-18 15:59:51,845 - evalscope - INFO - 
Benchmarking summary:
+-----------------------------------+-------------------------------------------------------------+
| Key                               | Value                                                       |
+===================================+=============================================================+
| Time taken for tests (s)          | 0.015                                                       |
+-----------------------------------+-------------------------------------------------------------+
| Number of concurrency             | 4                                                           |
+-----------------------------------+-------------------------------------------------------------+
| Total requests                    | 15                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Succeed requests                  | 0                                                           |
+-----------------------------------+-------------------------------------------------------------+
| Failed requests                   | 15                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Throughput(average tokens/s)      | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average QPS                       | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average latency (s)               | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average time to first token (s)   | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average time per output token (s) | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average input tokens per request  | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average output tokens per request | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average package latency (s)       | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average package per request       | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Expected number of requests       | 15                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Result DB path                    | ./outputs/20241218_155948/Qwen1.5-7B-Chat/benchmark_data.db |
+-----------------------------------+-------------------------------------------------------------+
2024-12-18 15:59:51,845 - evalscope - INFO - Too little data to calculate quantiles!
```

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
  - Euler 21.10 LTS

- Python版本 / Python Version:
  - [ ] 3.10


## 其他信息 / Additional Information

如果有其他相关信息，请在此处提供。 / If there is any other relevant information, please provide it here.


你请求的url不支持额外的参数请求，包括 seed 和 stream_options，你可以看一些怎么去修改模型服务的配置

是的，我的模型服务不接受seed和stream_options作为参数，这个只能修改模型服务吗？ 是否可以在evalscope这边做设置取消这两个参数呢？

请问你是用的什么框架启动的服务呢

是别人开发的一个自定义框架，服务端这边不是很方便去修改

如果不支持这两个参数的话，可以用下面方式启动（**注意：请拉取main分支代码安装**）
```
from evalscope.perf.main import run_perf_benchmark

task_cfg = {"url": "http://localhost:52415/v1/chat/completions",
            "parallel": 4,
            "model": "qwen/Qwen1.5-7B-Chat",
            "number": 15,
            "api": "openai",
            "dataset": "openqa",
            "stream": None,  # 不设置
            "seed": None。    # 不设置
}
run_perf_benchmark(task_cfg)
```

it worked!!

![aaf60da39c361db2b2e764ef0cfdacb](https://github.com/user-attachments/assets/4847e569-381b-482d-9ae6-a34d3e4fb08e)
不过存在一个问题，就是TPOT变成了nan这是什么原因呢？

tpot在stream为True下生效

If you find EvalScope helpful, please feel free to STAR our project. Thank YOU!
如果EvalScope对您有帮助，请给我们加个STAR鼓励，谢谢！