Repository: modelscope/evalscope
Issue: 为啥提高并发和请求数反而速度会变快，正常吗？
Root cause: Documentation Deficiency

为啥提高并发和请求数反而速度会变快？

命令1：evalscope perf --url 'http://0.0.0.0:8680/v1/chat/completions' --parallel 1 --model 'glm-4-9b-chat' --log-every-n-query 10 --read-timeout=120 --dataset-path './datasets/test.jsonl' -n 1 --max-prompt-length 32768 --api openai --stream --stop '
' --dataset openqa --debug

命令2：evalscope perf --url 'http://0.0.0.0:8680/v1/chat/completions' --parallel 10 --model 'glm-4-9b-chat' --log-every-n-query 10 --read-timeout=120 --dataset-path './datasets/test.jsonl' -n 10 --max-prompt-length 32768 --api openai --stream --stop '
' --dataset openqa --debug

命令1执行的结果：
Benchmarking summary:
 Time taken for tests: 10.006 seconds
 Expected number of requests: 1
 Number of concurrency: 1
 Total requests: 1
 Succeed requests: 1
 Failed requests: 0
 Average QPS: 0.100
 Average latency: 9.638
 Throughput(average output tokens per second): 74.655
 Average time to first token: 0.200
 Average input tokens per request: 741.000
 Average output tokens per request: 747.000
 Average time per output token: 0.01339
 Average package per request: 749.000
 Average package latency: 0.013
 Too little data to calculate quantiles!


命令2执行的结果：
Benchmarking summary:
 Time taken for tests: 43.093 seconds
 Expected number of requests: 10
 Number of concurrency: 10
 Total requests: 10
 Succeed requests: 10
 Failed requests: 0
 Average QPS: 0.232
 Average latency: 28.302
 Throughput(average output tokens per second): 342.375
 Average time to first token: 3.435
 Average input tokens per request: 1503.900
 Average output tokens per request: 1475.400
 Average time per output token: 0.00292
 Average package per request: 1477.400
 Average package latency: 0.017
 Percentile of time to first token:
     p50: 3.4349
     p66: 3.4351
     p75: 3.4352
     p80: 3.4362
     p90: 3.4363
     p95: 3.4363
     p98: 3.4363
     p99: 3.4363
 Percentile of request latency:
     p50: 32.0549
     p66: 32.2285
     p75: 32.6104
     p80: 35.4587
     p90: 42.5055
     p95: 42.5055
     p98: 42.5055
     p99: 42.5055

## 使用的工具 / Tools Used
- [ ] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend 
- [ ] RAGEval backend
- [√] Perf / 模型推理压测工具
- [ ] Arena  /竞技场模式

## 运行环境 / Runtime Environment
-VLLM运行glm-4-9b-chat,双4090

- 操作系统 / Operating System:
  - [ ] Windows
  - [ ] macOS
  - [√] Ubuntu

- Python版本 / Python Version:
  - [ ] 3.11
  - [√] 3.10
  - [ ] 3.9


第一个请求耗时可能会更长一些，这其中包括与服务器建立连接的时间，可以两个都设置`-n 10`再尝试

> 第一个请求耗时可能会更长一些，这其中包括与服务器建立连接的时间，可以两个都设置`-n 10`再尝试

evalscope perf --url 'http://.0.0.0.0:8680/v1/chat/completions' --parallel 1 --model 'glm-4-9b-chat' --log-every-n-query 10 --read-timeout=120 --dataset-path './datasets/test.jsonl' -n 10 --max-prompt-length 32768 --api openai --stream --stop '
' --dataset openqa --debug

Benchmarking summary:
 Time taken for tests: 177.201 seconds
 Expected number of requests: 10
 Number of concurrency: 1
 Total requests: 10
 Succeed requests: 10
 Failed requests: 0
 Average QPS: 0.056
 Average latency: 17.705
 Throughput(average output tokens per second): 77.105
 Average time to first token: 0.366
 Average input tokens per request: 1503.900
 Average output tokens per request: 1366.300
 Average time per output token: 0.01297
 Average package per request: 1368.300
 Average package latency: 0.013
 Percentile of time to first token:
     p50: 0.4228
     p66: 0.4367
     p75: 0.4409
     p80: 0.4981
     p90: 0.5522
     p95: 0.5522
     p98: 0.5522
     p99: 0.5522
 Percentile of request latency:
     p50: 19.1694
     p66: 20.6969
     p75: 22.2028
     p80: 23.3344
     p90: 30.7874
     p95: 30.7874
     p98: 30.7874
     p99: 30.7874

--parallel 1 时，请求一个一个发送；而--parallel 10，请求是10个一起发送，请确认使用的server是否有自动batch功能呢，可能一次处理多个请求

> 由于长时间未收到活动，我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！