Repository: modelscope/evalscope
Issue: evalscope - ERROR
Root cause: Configuration Error


启动命令，测试sglang  接口就有问题，vllm正常
evalscope perf     --url "http://10.192.128.28:8080/v1/chat/completions"        --model sg-r1     --parallel 1     --api openai     --dataset openqa  --st
ream  --number 15  --max-tokens 1024 --min-tokens 1024 

错误日志:
Processing: 0it [00:00, ?it/s]2025-03-06 05:11:35,638 - evalscope - ERROR - 
2025-03-06 05:11:35,638 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '盗贼天赋盗贼怎么加天赋?知道告诉一下下啦~~'}], 'model': 'sg-r1', 'max_tokens': 1024, 'min_tokens': 1024, 'seed': 42, 'stream': True, 'stream_options': {'include_usage': True}} failed, state_code: None, data: 
Processing: 1it [04:00, 240.37s/it]2025-03-06 05:15:36,629 - evalscope - ERROR - 
2025-03-06 05:15:36,629 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': 'Ps2比 画面好吗?差距很大吗? Ps2比psp画面好吗?差距很大吗?'}], 'model': 'sg-r1', 'max_tokens': 1024, 'min_tokens': 1024, 'seed': 42, 'stream': True, 'stream_options': {'include_usage': True}} failed, state_code: None, data: 
Processing: 2it [08:01, 240.74s/it]2025-03-06 05:19:37,633 - evalscope - ERROR - 
2025-03-06 05:19:37,633 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '加装方向助力我有一普桑想加装一套方向助力可以吗？？怎么改？'}], 'model': 'sg-r1', 'max_tokens': 1024, 'min_tokens': 1024, 'seed': 42, 'stream': True, 'stream_options': {'include_usage': True}} failed, state_code: None, data: 
Processing: 3it [12:02, 240.86s/it]2025-03-06 05:23:38,625 - evalscope - ERROR - 
2025-03-06 05:23:38,625 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '比斯巨兽的皮干什么用的？多少钱？'}], 'model': 'sg-r1', 'max_tokens': 1024, 'min_tokens': 1024, 'seed': 42, 'stream': True, 'stream_options': {'include_usage': True}} failed, state_code: None, data: 
Processing: 4it [16:03, 240.91s/it]2025-03-06 05:27:39,637 - evalscope - ERROR - 
2025-03-06 05:27:39,637 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '神州行手机号码成为空号了怎么办，卡里还有几十圆钱，还可以再用吗'}], 'model': 'sg-r1', 'max_tokens': 1024, 'min_tokens': 1024, 'seed': 42, 'stream': True, 'stream_options': {'include_usage': True}} failed, state_code: None, data: 
Processing: 5it [20:04, 240.95s/it]^B[^[[A^[[A^[[A^[[A^[[A2025-03-06 05:31:40,643 - evalscope - ERROR - 
2025-03-06 05:31:40,644 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '最近脸上长了好多脂肪粒，这个该怎么破？ 记得前段时间你好像也有～'}], 'model': 'sg-r1', 'max_tokens': 1024, 'min_tokens': 1024, 'seed': 42, 'stream': True, 'stream_options': {'include_usage': True}} failed, state_code: None, data: 
Processing: 6it [24:05, 240.97s/it]


你使用的是什么版本的evalscope和sglang，我这边测试没问题啊，你看看模型服务端有报错没

```shell
这个是版本
(evalscope) root@com1:~# pip list|grep evalscope
evalscope                     0.12.0
(SGLang_r1) root@compute01:~# pip list |grep sglang
sglang                            0.4.3.post2


sglang端日志:
python -m sglang.launch_server --model-path /root/DeepSeek-R1-AWQ/ --tp 8  --host 0.0.0.0 --port 8080 --disable-cuda-graph --mem-fraction-static 0.9 --served-model-name  sg-r1 --dtype float16  --trust-remote-code

[2025-03-06 14:12:31 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 3, cache hit rate: 60.83%, token usage: 0.00, #running-req: 0, #queue-req: 0                   
[2025-03-06 14:12:40 TP0] Decode batch. #running-req: 1, #token: 1, token usage: 0.00, gen throughput (token/s): 0.05, #queue-req: 0                                               
[2025-03-06 14:12:40] INFO:     10.192.128.34:37722 - "POST /v1/chat/completions HTTP/1.1" 200 OK                                                                                  
[2025-03-06 14:12:45] INFO:     10.192.128.34:55838 - "POST /v1/chat/completions HTTP/1.1" 200 OK                                                                                  
[2025-03-06 14:12:45 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 624, cache hit rate: 68.77%, token usage: 0.00, #running-req: 0, #queue-req: 0                 
[2025-03-06 14:13:28 TP0] Decode batch. #running-req: 1, #token: 665, token usage: 0.00, gen throughput (token/s): 0.85, #queue-req: 0                                             
[2025-03-06 14:14:11 TP0] Decode batch. #running-req: 1, #token: 705, token usage: 0.00, gen throughput (token/s): 0.92, #queue-req: 0                                             
[2025-03-06 14:14:49 TP0] Decode batch. #running-req: 1, #token: 745, token usage: 0.00, gen throughput (token/s): 1.06, #queue-req: 0                                             
[2025-03-06 14:15:23 TP0] Decode batch. #running-req: 1, #token: 785, token usage: 0.00, gen throughput (token/s): 1.15, #queue-req: 0                                             
[2025-03-06 14:15:58 TP0] Decode batch. #running-req: 1, #token: 825, token usage: 0.00, gen throughput (token/s): 1.14, #queue-req: 0                                             
[2025-03-06 14:16:33 TP0] Decode batch. #running-req: 1, #token: 865, token usage: 0.00, gen throughput (token/s): 1.15, #queue-req: 0

我使用下面的命令测试 0.4.3.post3 版本的sglang，没有出现你上述的问题：
```shell
CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --model-path models/Qwen2.5-0.5B-Instruct --served-model-name Qwen2.5-0.5B-Instruct --port 8801 --host 0.0.0.0
```
```shell
evalscope perf \
 --parallel 6 \
 --url http://127.0.0.1:8801/v1/chat/completions \
 --model Qwen2.5-0.5B-Instruct \
 --log-every-n-query 5 \
 --connect-timeout 6000 \
 --read-timeout 6000 \
 --max-tokens 1024 \
 --min-tokens 1024 \
 --api openai \
 --dataset openqa \
 -n 30 \
 --stream
```

出现同样类似问题，找不到原因何在


2025-03-07 00:05:56,560 - evalscope - ERROR - 
2025-03-07 00:05:56,560 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '步步高股价为什么开盘价是20多，昨天不还是50多呢么'}], 'model': 'QwQ-32B', 'max_tokens': 2048, 'seed': 42, 'stream': True, 'stream_options': {'include_usage': True}} failed, state_code: None, data: 
2025-03-07 00:05:56,561 - evalscope - ERROR - 
2025-03-07 00:05:56,561 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '满满一杯水，不用吸管，怎样才能喝到杯底的水？'}], 'model': 'QwQ-32B', 'max_tokens': 2048, 'seed': 42, 'stream': True, 'stream_options': {'include_usage': True}} failed, state_code: None, data: 
Processing: 222it [06:10,  2.16s/it]2025-03-07 00:05:59,794 - evalscope - ERROR - 
2025-03-07 00:05:59,794 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '关于木宝箱的问题大家说说是闯关得的好还是在魔岭拿的木宝箱好?'}], 'model': 'QwQ-32B', 'max_tokens': 2048, 'seed': 42, 'stream': True, 'stream_options': {'include_usage': True}} failed, state_code: None, data: 
Processing: 225it [06:11,  2.16s/it]2025-03-07 00:06:01,560 - evalscope - ERROR - 
2025-03-07 00:06:01,560 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '坐着或蹲着时间久了再站起来,头会晕,眼会黑一会,这是不是贫血?'}], 'model': 'QwQ-32B', 'max_tokens': 2048, 'seed': 42, 'stream': True, 'stream_options': {'include_usage': True}} failed, state_code: None, data: 
2025-03-07 00:06:03,560 - evalscope - ERROR - 
2025-03-07 00:06:03,560 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '每天清晨起来口干口苦，唾液都是黄色，什么原因'}], 'model': 'QwQ-32B', 'max_tokens': 2048, 'seed': 42, 'stream': True, 'stream_options': {'include_usage': True}} failed, state_code: None, data: 
2025-03-07 00:06:04,560 - evalscope - ERROR - 
2025-03-07 00:06:04,561 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '电脑从硬盘中读取一个文件的过程是怎样的？'}], 'model': 'QwQ-32B', 'max_tokens': 2048, 'seed': 42, 'stream': True, 'stream_options': {'include_usage': True}} failed, state_code: None, data: 


---
 evalscope perf --model QwQ-32B --url http://192.168.60.233/v1/chat/completions --api openai --api-key sk-xxxxxx --parallel 200 --number 600 --dataset openqa --stream


麻烦启动参数里面加上--debug，看一下打印的log

看了一下，可能是推理比较慢，请求超时了，可以设置更长的timeout参数。例如：
```bash
 --connect-timeout 60000 \
 --read-timeout 60000 \
```
这块超时逻辑我也修改一下