Repository: vllm-project/guidellm
Issue: Does guidellm support benchmarking for SGlang
Root cause: Unimplemented Feature Gap

I am using Guidellm to benchmarking SGLang and vllm (both of their API is openAI compatible) on A800.
vllm runs very smoothly. SGlang return some errors and does not generate complete report.
```
sglang version  0.3.4.post2   
```
My SGlang start up settings is
```
export MKL_SERVICE_FORCE_INTEL=1
python -m sglang.launch_server --model-path deepseek-ai/deepseek-llm-7b-chat --port 30000 --log-level debug --log-requests
```

And My guidellm script is
```
guidellm \
  --target "http://localhost:30000/v1" \
  --model "deepseek-ai/deepseek-llm-7b-chat" \
  --data-type emulated \
  --data "prompt_tokens=512,generated_tokens=128" \
  --rate-type "synchronous"
```

The result have warning at beginning and have exception and coredump at the end.

```
╭─ Benchmarks ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ [10:13:41]   100% synchronous   (24.66 req/sec avg)                                                                                                                                                                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
  Generating report... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ (1/1) [ 0:01:59 < 0:00:00 ]
2024-10-29T10:15:41.213489+0800 | percentiles | WARNING - No data points available to calculate percentiles.
2024-10-29T10:15:41.216092+0800 | percentiles | WARNING - No data points available to calculate percentiles.
2024-10-29T10:15:41.217926+0800 | mean | WARNING - No data points available to calculate mean.
2024-10-29T10:15:41.219155+0800 | mean | WARNING - No data points available to calculate mean.
╭─ GuideLLM Benchmarks Report (stdout) ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ ╭─ Benchmark Report 1 ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │
│ │ Backend(type=openai_server, target=http://localhost:30000/v1, model=deepseek-ai/deepseek-llm-7b-chat)                                                                                                                                                                    │ │
│ │ Data(type=emulated, source=prompt_tokens=512,generated_tokens=128, tokenizer=deepseek-ai/deepseek-llm-7b-chat)                                                                                                                                                           │ │
│ │ Rate(type=synchronous, rate=None)                                                                                                                                                                                                                                        │ │
│ │ Limits(max_number=None requests, max_duration=120 sec)                                                                                                                                                                                                                   │ │
│ │                                                                                                                                                                                                                                                                          │ │
│ │                                                                                                                                                                                                                                                                          │ │
│ │ Requests Data by Benchmark                                                                                                                                                                                                                                               │ │
│ │ ┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓                                                                                                                                                                               │ │
│ │ ┃ Benchmark   ┃ Requests Completed ┃ Request Failed ┃ Duration   ┃ Start Time ┃ End Time ┃                                                                                                                                                                               │ │
│ │ ┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩                                                                                                                                                                               │ │
│ │ │ synchronous │ 2958/2958          │ 0/2958         │ 119.94 sec │ 10:13:41   │ 10:15:41 │                                                                                                                                                                               │ │
│ │ └─────────────┴────────────────────┴────────────────┴────────────┴────────────┴──────────┘                                                                                                                                                                               │ │
│ │                                                                                                                                                                                                                                                                          │ │
│ │ Tokens Data by Benchmark                                                                                                                                                                                                                                                 │ │
│ │ ┏━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓                                                                                                                                                                   │ │
│ │ ┃ Benchmark   ┃ Prompt ┃ Prompt (1%, 5%, 50%, 95%, 99%)    ┃ Output ┃ Output (1%, 5%, 50%, 95%, 99%) ┃                                                                                                                                                                   │ │
│ │ ┡━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩                                                                                                                                                                   │ │
│ │ │ synchronous │ 512.85 │ 512.0, 512.0, 512.0, 515.0, 516.0 │ 0.00   │ 0.0, 0.0, 0.0, 0.0, 0.0        │                                                                                                                                                                   │ │
│ │ └─────────────┴────────┴───────────────────────────────────┴────────┴────────────────────────────────┘                                                                                                                                                                   │ │
│ │                                                                                                                                                                                                                                                                          │ │
│ │ Performance Stats by Benchmark                                                                                                                                                                                                                                           │ │
│ │ ┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓                                                                       │ │
│ │ ┃ Benchmark   ┃ Request Latency [1%, 5%, 10%, 50%, 90%, 95%, 99%] (sec) ┃ Time to First Token [1%, 5%, 10%, 50%, 90%, 95%, 99%] (ms) ┃ Inter Token Latency [1%, 5%, 10%, 50%, 90% 95%, 99%] (ms) ┃                                                                       │ │
│ │ ┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩                                                                       │ │
│ │ │ synchronous │ 0.03, 0.03, 0.03, 0.04, 0.05, 0.05, 0.06                │ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0                          │ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0                         │                                                                       │ │
│ │ └─────────────┴─────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────┘                                                                       │ │
│ │                                                                                                                                                                                                                                                                          │ │
│ │ Performance Summary by Benchmark                                                                                                                                                                                                                                         │ │
│ │ ┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓                                                                                                                                            │ │
│ │ ┃ Benchmark   ┃ Requests per Second ┃ Request Latency ┃ Time to First Token ┃ Inter Token Latency ┃ Output Token Throughput ┃                                                                                                                                            │ │
│ │ ┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩                                                                                                                                            │ │
│ │ │ synchronous │ 24.66 req/sec       │ 0.04 sec        │ 0.00 ms             │ 0.00 ms             │ 0.00 tokens/sec         │                                                                                                                                            │ │
│ │ └─────────────┴─────────────────────┴─────────────────┴─────────────────────┴─────────────────────┴─────────────────────────┘                                                                                                                                            │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
FATAL: exception not rethrown
Aborted (core dumped)
```

I think the requests per second (req/sec) are too high to be true. After checking the logs of sglang, I found that most of the requests are empty, and only a few requests show up at intervals.

My question is does guidellm support SGlang. and if it does how to address this issue.

![image](https://github.com/user-attachments/assets/fc4c5c1e-07c3-4b73-a736-6e139d24fa78)





 


Hi @jbymy2000, SGlang currently is not supported natively by GuideLLM. We initially built GuideLLM to surface benchmarks for vLLM only as that is the most popular OSS inference server that we are seeing be used. 

SGLang support is something we can add to our GuideLLM backlog, but in full transparency, we have slowed down further development on GuideLLM to focus on other company initiatives at this time. 

This would also make a great community contribution to help us expand the number of supported inference servers eg. SGlang, TensorRT-LLM, TGI...