zhiminy/EvalEng huggingface dataset:
Repository: vllm-project/guidellm
Issue: formatting json file with benchmark results
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
It is not convenient to view a json file with benchmark results. Everything is in 1 line, you have to install additional extensions to parse or resave the file.

**Describe the solution you'd like**
Formatting when saving a json file.

**Describe alternatives you've considered**
installing prettifyjson or resaving the file.
I would also suggest considering the option of splitting files: with prompta and model responses and with metrics, as llmperf does.

**Additional context**
```python
def save_json(data, file_to_save):
    with open(file_to_save, 'w', encoding='utf-8') as file:
        json.dump(data, file, ensure_ascii=False, indent=4)
```


Fixed by #372
Repository: vllm-project/guidellm
Issue: guidance_report.json from default flow is very large
Root cause: Unimplemented Feature Gap

After running the default flow on Mistral in vLLM, there is a large (>100MB) report json in the directory I ran the commands. This seems quite heavy-weight, especially for a json file. 

Instead, I would expect there to be a light csv or something summarizing the report that is displayed in the terminal so I can easily load my results into a spreadsheet or pandas dataframe for post-processing.

Commands:
```
vllm serve mistralai/Mistral-7B-Instruct-v0.3
guidellm --target "http://localhost:8000/v1" --model mistralai/Mistral-7B-Instruct-v0.3
```

Directory output:
```bash
guidellm git:(main) ll
total 114M
-rw-rw-r-- 1 mgoin mgoin 3.4K Aug 27 15:56 CODE_OF_CONDUCT.md
-rw-rw-r-- 1 mgoin mgoin 2.4K Aug 27 15:56 CONTRIBUTING.md
-rw-rw-r-- 1 mgoin mgoin 7.1K Aug 27 15:56 DEVELOPING.md
drwxrwxr-x 4 mgoin mgoin   95 Aug 27 15:56 docs
-rw-rw-r-- 1 mgoin mgoin 114M Aug 27 16:19 guidance_report.json <<
-rw-rw-r-- 1 mgoin mgoin  12K Aug 27 15:56 LICENSE
-rw-rw-r-- 1 mgoin mgoin   16 Aug 27 15:56 MANIFEST.in
-rw-rw-r-- 1 mgoin mgoin 7.3K Aug 27 15:56 pyproject.toml
-rw-rw-r-- 1 mgoin mgoin  11K Aug 27 15:56 README.md
drwxrwxr-x 4 mgoin mgoin   59 Aug 27 15:56 src
drwxrwxr-x 6 mgoin mgoin  104 Aug 27 15:56 tests
-rw-rw-r-- 1 mgoin mgoin 1.7K Aug 27 15:56 tox.ini
drwxrwxr-x 2 mgoin mgoin   66 Aug 27 15:56 utils
```

We have opened a ticket to adjust how we handle this and trim what we save. For now, though, I've defaulted it not to save the output file and instead rely on the stdout.

Is there a way to output the results into a simple table or csv for pasting into a spreadsheet?

Yes, printing out to CSV will be included as part of that work

Fixing the original complaint about the size with #91. There will be a separate task for general report output types which will be covered by a separate issue
Repository: vllm-project/guidellm
Issue: Online docs are not updating
Root cause: Configuration Error


### Discussed in https://github.com/vllm-project/guidellm/discussions/396

<div type='discussions-op-text'>

<sup>Originally posted by **tukwila** October  9, 2025</sup>
I found guidellm benchmark test example pull request (https://github.com/vllm-project/guidellm/pull/328) is merged into main but not published in https://blog.vllm.ai/guidellm/main/examples/; what can i do to complete this demo content？ </div>

@sjmonson 
I tried to find out the problem and the following is result:
1. ci job: cleanup-ui-pr-preview in action: development-cleanup.yml is just for ui commit except docs commit; so i'm not sure if this job should be corrected, for example:
https://github.com/vllm-project/guidellm/actions/runs/18418700153/job/52488292661;

2. anther job: ui-pr-preview in development.yml is the same:
https://github.com/vllm-project/guidellm/actions/runs/18161932896/job/51694675090?pr=328

maybe these jobs are reasonable, so i will submit totally new ci job as bug fix to update gh-pages branch. If any problem, pls correct me. :)
Repository: vllm-project/guidellm
Issue: Empty prompts at prompt length 1 while benchmarking DeepSeekV3
Root cause: Validation Gap

I'm seeing empty prompts when benchmarking DeepSeekV3 with prompt length 1.
I could not reproduce this with either `Qwen/Qwen3-30B-A3B-FP8` or `deepseek-ai/DeepSeek-V2-Lite`

This is the error I see coming out of guidellm
```
25-10-13 21:00:46|ERROR            |guidellm.backend.openai:text_completions:307 - OpenAIHTTPBackend request with headers: {'Content-Type': 'application/json'} and params: {} and payload: {'prompt': '', 'model': 'deepseek-ai/DeepSeek-V3', 'stream': True, 'stream_options': {'include_usage': True}, 'max_tokens': 100, 'stop': None, 'ignore_eos': True} failed: Client error '400 Bad Request' for url 'http://127.0.0.1:8192/v1/completions'
```

The vLLM server then logs:
```
(APIServer pid=3766874) INFO:     127.0.0.1:45320 - "POST /v1/completions HTTP/1.1" 400 Bad Request
```


**To Repro:**
```
vllm serve deepseek-ai/DeepSeek-V3 --hf-overrides.num_hidden_layers=4 --load-format=dummy --port 8192 -tp 2
```

```
guidellm benchmark \
      --target http://127.0.0.1:8192 \
      --rate-type constant \
      --rate 2048 \
      --max-requests 10000 \
      --data '{"prompt_tokens": 1, "output_tokens": 100}'
```


This issue should be fixed in the v0.4.0 refactor. I verified with an RC build from @sjmonson 

Fixed in v0.4.0
Repository: vllm-project/guidellm
Issue: Sample prompts not displaying correctly
Root cause: Algorithmic Error

**Describe the bug**
The sample prompts section should show prompt snippets used in the benchmark run. A recent update has changed some of the underlying code structure and now the request_args holds much more than the prompt. The prompt is not surfaced in the shown sample.

Visualized (this will go down eventually when the PR is closed): https://blog.vllm.ai/guidellm/ui/pr/427/

<img width="460" height="461" alt="Image" src="https://github.com/user-attachments/assets/c9a59f7a-705e-4a4e-986b-3690a8a1d87e" />

**Expected behavior**
Example of it working (this will go down eventually as well, but for now holds the previous working state) https://blog.vllm.ai/guidellm/ui/pr/391/ 

<img width="455" height="465" alt="Image" src="https://github.com/user-attachments/assets/bb04a9ff-d594-46a5-bae5-8216d2d7bf73" />

**Additional context**
The change comes from an update to what request_args holds in the benchmark report. Previously the UI data relied on request_args holding the prompt value, but now the prompt text is nested within. Something like: `request_args.method.body.model.messages[0].content[0].text`

See: https://github.com/vllm-project/guidellm/blob/main/src/guidellm/presentation/data_models.py#L120



Same issue with wrong location

<img width="570" height="329" alt="Image" src="https://github.com/user-attachments/assets/50ee2928-563a-4526-a8ca-6c614baf414d" />
Repository: vllm-project/guidellm
Issue: Define workload characteristics for different application/enterprise use cases
GitHub labels: documentation

**What is the URL, file, or UI containing proposed doc change**
Where does one find the original content or where would this change go?

[Service Level Objectives (SLOs) and Service Level Agreements (SLAs)](https://github.com/vllm-project/guidellm/blob/main/docs/service_level_objectives.md)

**What is the current content or situation in question**
This is a really useful document which describes recommended SLO targets for different enterprise use cases. Many times downstream customers have trouble defining key performance metrics for their desired application, so this is a great starting point. 

**What is the proposed change**
In addition to SLO targets, it will be helpful to suggest workload metrics that map to real-world applications. For example, 

- input and output token lengths 
- arrival rates 
- concurrency 
- prefix hit rate

These metrics might be described in terms of ranges, distributions, or averages. There is currently no standard that seems to be out there, so if we can work together and propose an initial starting point, that would be super useful. 

**Additional context**
There are documents that have been compiled that aim to do workload patterns + SLOs -> application mapping. Though incomplete, we can certainly use them as starting points. 

- [ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production](https://arxiv.org/abs/2505.09999)
- [LLM Workloads from llm-d](https://docs.google.com/document/u/2/d/1Ia0oRGnkPS8anB4g-_XPGnxfmOTOeqjJNb32Hlo_Tp0/edit?tab=t.0#heading=h.7dpbklgr4met)
- [Project Compass from Red Hat](https://github.com/redhat-et/compass/blob/main/data/demo_scenarios.json)



There's also: https://mlcommons.org/benchmarks/inference-datacenter/

Thanks for the suggestions @jgchn, we have support to add in schema files that would predefine these. I'll attach this to the feature request that is ongoing for that
Repository: vllm-project/guidellm
Issue: failed requests results in low throughput and high TTFT/ITL
Root cause: Validation Gap

**Describe the bug**
when benchmark using guidellm, some failed requests results in low throughput and high TTFT/ITL

**Expected behavior**
A clear and concise description of what you expected to happen.
throughput, TTFT/ITL should filter results of failed requests.

**Environment**
Include all relevant environment information:
1. OS [e.g. Ubuntu 20.04]:Ubuntu 22.04
2. Python version [e.g. 3.12.2]: Python 3.12.12
guidellm: 0.3.1

**To Reproduce**
benchmark script:
```
guidellm benchmark \
  --target "http://localhost:30000" \
  --model DeepSeek-V3.1-Terminus \
  --processor DeepSeek-V3.1-Terminus \
  --data-sampler "random" \
  --data "prompt_tokens=4000,output_tokens=512,prompt_tokens_stdev=200,output_tokens_stdev=50" \
  --rate-type poisson \
  --rate 3    \
  --max-requests 1024
```
Repository: vllm-project/guidellm
Issue: serializer.deserialize(serialized) duplicate the last dict data
Root cause: Algorithmic Error

**Describe the bug**
A clear and concise description of what the bug is.

when i debug xfail ut case: tests/unit/utils/test_encoding.py::TestMessageEncoding::test_encode_decode_generative[dict_serialization_msgpack-obj0]
assertion failure is:

<img width="2173" height="478" alt="Image" src="https://github.com/user-attachments/assets/4557e261-c813-44ae-9e43-5a49f5f7a1e4" />

As screenshot show, the last dict data total_tokens=None is duplicated.

And the problem is serializer.deserialize(serialized) in src/guidellm/utils/encoding.py:

<img width="2196" height="1393" alt="Image" src="https://github.com/user-attachments/assets/5a738080-8d4b-4690-9ad8-e42672507448" />

**Expected behavior**
A clear and concise description of what you expected to happen.

**Environment**
Include all relevant environment information:
1. OS [e.g. Ubuntu 20.04]: conda 25.3.1
2. Python version [e.g. 3.12.2]: guidellm version: 0.4.0.dev0

**To Reproduce**
Exact steps to reproduce the behavior:


**Errors**
If applicable, add a full print-out of any errors or exceptions that are raised or include screenshots to help explain your problem.

**Additional context**
Add any other context about the problem here. Also include any relevant files.


@tukwila just confirming that this is happening on main/0.4?

> [@tukwila](https://github.com/tukwila) just confirming that this is happening on main/0.4?

Yes, now in main:
guidellm version: 0.5.0.dev10

<img width="2039" height="1302" alt="Image" src="https://github.com/user-attachments/assets/b7d05edd-7c99-4be3-a123-b11d25aa8f8a" />

but we cannot find a solution till now:
<img width="934" height="376" alt="Image" src="https://github.com/user-attachments/assets/24badea1-55af-4355-9e2a-622ba05422cf" />
Repository: vllm-project/guidellm
Issue: [Usage] Per-strategy max_requests constraints not respected in Sweep Mode
Root cause: Algorithmic Error

**Describe the bug**  
When using Sweep Mode with a custom profile that specifies a list of `max_requests` values under `constraints`, all sub-strategies appear to use the same `max_requests` value instead of the respective value from the list.

**Purpose / Expected Behavior**  
I would like to assign a different `max_requests` value to each strategy generated during a sweep (e.g., for fine-grained control over benchmark duration or resource usage per point in the sweep).

**Environment**  
- GuideLLM version: 0.4.0-dev
- Python version: 3.12
- Backend: vLLM 

**To Reproduce**  
1. Clone and install GuideLLM:
   ```bash
   git clone https://github.com/vllm-project/guidellm.git
   cd guidellm
   pip install -e .
   ```

2. Create a profile file `myProfile.json`:
   ```json
   {
     "profile": {
       "type_": "sweep",
       "sweep_size": 5,
       "constraints": {
         "max_requests": [10, 100, 100, 100, 100]
       }
     },
     "target": "http://localhost:8000",
     "data": [
       "prompt_tokens=512,prompt_tokens_stdev=128,prompt_tokens_min=1,prompt_tokens_max=1024,output_tokens=256,output_tokens_stdev=64,output_tokens_min=1,output_tokens_max=1024"
     ],
     "output_formats": ["benchmarks_json.html"]
   }
   ```

3. Run the benchmark:
   ```bash
   guidellm benchmark run --scenario src/guidellm/benchmark/scenarios/myProfile.json
   ```

**Observed Behavior**  
All five sweep strategies show the same `max_requests` value (all use 10), as shown in the output table:

![Sweep strategies all using same max_requests](https://github.com/user-attachments/assets/aba4ee4d-4e39-4c82-9b57-7087920c49b1)

This suggests the per-strategy constraint in `constraints.max_requests` is not being applied correctly.



**Additional Context**  
- Is the `constraints` field documented for per-strategy configuration in sweep mode?  

Thanks @AiKiAi-stack, adding this on as a general feature request. Let us know if you're interested in contributing/working on it, otherwise we'll work on getting this typed up and added to the roadmap

Thanks! Yes, I'm interested in contributing and would like to work on this feature. 

See also #253
Repository: vllm-project/guidellm
Issue: Request: Include vLLM request ID in GuideLLM benchmark results
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
When checking the GuideLLM results file, I noticed that the request ID does not match the one returned by vLLM. Could GuideLLM output the actual vLLM request ID? This is important for joining GuideLLM results with vLLM responses and other telemetry data, such as distributed traces.

**Describe the solution you'd like**
Add support for including the actual vLLM request ID in GuideLLM’s output.


CC @markurtz
Repository: vllm-project/guidellm
Issue: v0.3.1 container includes v0.4.0.dev0 GuideLLM python package
Root cause: Configuration Error

**Describe the bug**
The v0.3.1 container contains the v0.4.0.dev0 version of GuideLLM.

**Expected behavior**
Container tag should match the version of GuideLLM that it includes

**Environment**
N/A

**To Reproduce**
```
podman pull ghcr.io/vllm-project/guidellm:v0.3.1
podman run -it ghcr.io/vllm-project/guidellm:v0.3.1 --version
```

Output
```
guidellm version: 0.4.0.dev0
```

**Errors**
If applicable, add a full print-out of any errors or exceptions that are raised or include screenshots to help explain your problem.

**Additional context**
Add any other context about the problem here. Also include any relevant files.


This is a CI bug. v0.4.0.dev0 is equivalent to v0.3.1.

@sjmonson is this something we can close out / what's the intended plan?
Repository: vllm-project/guidellm
Issue: How can we dump detailed ttft and token generated time data for all benchmared requests?
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
No

**Describe the solution you'd like**
How can we dump detailed ttft and token generated time data for all benchmared requests?
Thus we can do some customized postprocess and deep analysis.
Repository: vllm-project/guidellm
Issue: Request repetition across sweep rounds leads to 100% prefix-hit ratio (possible bug)
Root cause: Algorithmic Error

**Describe the bug**
We noticed that request payloads are repeating across sweep rounds when running a benchmark with `rate-type=sweep`.

Example configuration:
- max_req = 5
- rate-type = sweep
- rates = [sync, tput, 2 req/s, 4 req/s] (4 rounds total)
- Single benchmark 


Observed Behavior: Within a single benchmark run, the same 5 requests are reused in every sweep iteration. For instance:
- Iteration 1 (sync): req0, req1, req2, req3, req4
- Iteration 2 (tput): req0, req1, req2, req3, req4
- Iteration 3 (2 req/s): req0, req1, req2, req3, req4
- Iteration 4 (4 req/s): req0, req1, req2, req3, req4

This repetition continues for all rounds.

Because the same requests are reused after the first iteration, the prefix cache is fully warm, resulting in a 100% prefix-hit ratio. This makes comparisons across sweep rates unfair,


**Expected behavior**


Each sweep iteration should ideally use unique requests, or there should be an option to disable request reuse. This would avoid prefix-cache artifacts and allow accurate measurement of prefill performance across rate variations.
Repository: vllm-project/guidellm
Issue: how to show mean and p99 TTFT/ITL for guidellm 0.4.0? Can only see Mdn   | p95  in console output
Root cause: Unimplemented Feature Gap

how to show mean and p99 TTFT/ITL for guidellm 0.4.0? Can only see Mdn   | p95  in console output

Can we support config which types metrics (mean, median, p95 p99, etc. ) to show in outputs, but not hard coded (in add_stats
 default parameter types: Sequence[StatTypesAlias] = ("median", "p95"))?  Thanks.

The console output is only for giving a broad overview. If you want more detailed metrics. Please parse them from the `json` output which goes into significantly more detail.

@llc-kc, I'll cahnge this over to a feature request for this. Currently, there is no support in console but the output formats including csv and json have that information, as @sjmonson mentioned, but changing the stats in the console output seems reasonable. There is limited space, though, so console we will only be able to enable two metrics realistically without the tables becoming too large.
Repository: vllm-project/guidellm
Issue: [Rate Type] Concurrencies
Root cause: Unimplemented Feature Gap

Hello, 

I am trying to integrate `guidellm` into a benchmark suite. And there we ran different load tests based on use concurrencies. We define user concurrenies as "users" that send requests after each other. Meaning send request -> wait for response -> send next request. 

I first assumed that's what is done with "constant" and "rate" but there is send way more requests as they are send per second. Is there a way to customize the "user concurrency"? I assume that concurrency == synchronous type. But would be create if i could do something like 

```bash
guidellm --target "http://localhost:8080/v1" --model "meta-llama/Meta-Llama-3.1-8B-Instruct"  --data-type emulated --data "prompt_tokens=550,generated_tokens=250" --max-seconds 60 --rate-type concurrent --rate 1 --rate 2 --rate 10 --rate 50 --output-path r.json
```


Hey @philschmid, I understand what you mean about this request. You'd specifically like to be able to keep a fixed number of concurrent requests over the life of the benchmark where as soon as one finishes it immediately starts a new one, is that correct? You can't easily figure out currently through the constant or poison rate types since those are set as the number of requests per second rather which you'd have to adjust those until you hit the average number of concurrent users, right?

Hey, 

Yes. I am looking for a way to benchmark the load under e,g, 1, 2, 4, 8, 16, 32, 64, 128 concurrent users (send request -> wait for response, send again). 

But looking into more benchmarks and dashboard, people seem to switch to QPS (what `rate` should cover). So not sure how important this is. 

@philschmid support for this will be landing with #96 

Closing this out as this has landed now
Repository: vllm-project/guidellm
Issue: Authentication question
Root cause: Documentation Deficiency

This is a question that I can't seem to find an answer for.

My API requires authentication. How could I benchmark it with guidellm?

```
guidellm benchmark [run]... --headers "Authorization: Bearer xxx"



Usage: guidellm benchmark run [OPTIONS]
Try 'guidellm benchmark run --help' for help.

Error: No such option: --header
```



Found by reviewing the code:

```
export GUIDELLM__OPENAI__API_KEY=xxx
```
Repository: vllm-project/guidellm
Issue: index.md format is not right
GitHub labels: bug

**Describe the bug**
A clear and concise description of what the bug is.
In the following docs, key sections list format is not right:

1. docs/index.md

<img width="1067" height="1502" alt="Image" src="https://github.com/user-attachments/assets/860b00f0-fc37-4a01-9b06-002657cfc9c7" />

2. docs/guides/index.md

<img width="1698" height="1556" alt="Image" src="https://github.com/user-attachments/assets/bbe5f4c9-fd7e-42c9-9f8f-046f8c2b5ee9" />

3. docs/getting-started/index.md

<img width="1698" height="1225" alt="Image" src="https://github.com/user-attachments/assets/11070863-d8c2-4a33-976a-bd050b7445f6" />

4. docs/developer/index.md

<img width="1678" height="1150" alt="Image" src="https://github.com/user-attachments/assets/e3547d44-f836-4e07-a291-3173121d3f46" />


**Expected behavior**
A clear and concise description of what you expected to happen.

**Environment**
Include all relevant environment information:
1. OS [e.g. Ubuntu 20.04]: Monterey
2. Python version [e.g. 3.12.2]: Python 3.11.5
3. guidellm version: 0.5.0.dev10

**To Reproduce**
Exact steps to reproduce the behavior:


**Errors**
If applicable, add a full print-out of any errors or exceptions that are raised or include screenshots to help explain your problem.

**Additional context**
Add any other context about the problem here. Also include any relevant files.


This is intentional as the docs are in [MkDocs](https://www.mkdocs.org/) format.
Repository: vllm-project/guidellm
Issue: Pylock tooling does not explicitly lock platform
Root cause: Configuration Error

**Problem**

`pdm` is not properly configured to lock for each supported platform, thus the lockfile that is generated depends on the developer's machine.

**Solution**

Add explicit split-locks for each supported platform (Linux, MacOS) to `scripts/generate_pylock.sh` and regenerate the lockfile using these splits.

There is some strange behavior going on when trying to split the lock across platforms. When doing:

```sh
pdm lock --platform linux --python "~=3.12" $update_stratagy
pdm lock --platform macos --python "~=3.12" $update_stratagy --append
pdm lock --platform linux --python "<3.12" $update_stratagy --append
pdm lock --platform macos --python "<3.12" $update_stratagy --append
```

some of the locked packages use the Linux wheels while others use MacOS; instead of having separate markers for each.

Superseded by #483