zhiminy/EvalEng huggingface dataset:
Repository: vllm-project/guidellm
Issue: [FeatureReuqest] Auto-sweep with model saturation detection
Root cause: Unimplemented Feature Gap

**Description**
When performing a sweep add the ability to perform auto-sweep.
An auto-sweep runs through the different benchmarks while constantly checking for model saturation and shutting down when saturation is reached.

**Motivation**
GPU resources are expensive and we wouldn't like to keep them up longer then absolutely necessary. 
Moreover, one of the major goals of `guidellm` is to load test LLMs. 
Hence, once we've determined the maximum load the model can handle, i.e reached saturation, we can stop running benchmarks cause we already know the max RPS it can handle. 

**Saturation Detection**
When a model is saturated it does not finish current running requests. Hence, assuming all request are of similar size, if we periodically measure currently-running requests and we detect a large enough upwards trend it indicates a saturation. 

This check should start after a configurable delta after beginning the benchmark because initially it makes sense that there would be an upwards trend of concurrent requests.

@AlonKellner-Jounce - let's discuss this methodology further. where did we end up last we spoke on this. cc @markurtz 

Closing this out since we have the standard sweep pathway with throughput and now the new auto saturation detection landing soon
Repository: vllm-project/guidellm
Issue: [Feature Request] Measure request dispatch accuracy
Root cause: Unimplemented Feature Gap

**Description**
When using `constant` or `poisson` rates we would like to know how accurate the dispatch of requests really is.
So for each request we would like to have `scheduled_time` and actual `sent_time` as well as part of a summarized report file. 

For example, if we have a constant rate of `1 RPS` it would be meaningful to know that a request was sent after `1.2` seconds instead of `1` second.

**Motivation**
When analyzing the benchmarks in order to make decision regarding production grade workloads we need to know how accurate the measurement was. If the data is very skewed it would be wrong to make decisions based on it. Such problems are expected to arise under high load which makes it even more crucial to investigate as this is where systems typically fail.

Closing this out now that #96 has landed on main. Each request stored in the output now contains a "scheduler_info" object which contains the request times here along with any other important times for the benchmarking process.
Repository: vllm-project/guidellm
Issue: [Nice to Have] Container builds
Root cause: Unimplemented Feature Gap

It would be nice to have packages as container images to quickly get up to using guidellm and also reduces the volume of potential issues faced by users if we have a ready to go container available. 

Ref: [llm-load-test](https://github.com/orgs/openshift-psap/packages?repo_name=llm-load-test)

Relates to #119
Repository: vllm-project/guidellm
Issue: [Bug] Possibly incorrect output dump ? (The actual data appears to be fine)
Root cause: Algorithmic Error

```
cat fp8_fp8_tp2.json | jq '.benchmarks[] | .args.profile.measured_rates'
[
  0.020927754394140415,
  1.277633358306354,
  0.16187661325946062,
  0.3006715892920996,
  0.42809462336187576,
  0.5790540149106302,
  0.706221702530327,
  0.8352987384060198,
  0.9707840531576964,
  1.0963631997951027
]
[
  0.020927754394140415,
  1.277633358306354,
  0.16187661325946062,
  0.3006715892920996,
  0.42809462336187576,
  0.5790540149106302,
  0.706221702530327,
  0.8352987384060198,
  0.9707840531576964,
  1.0963631997951027
]
[
  0.020927754394140415,
  1.277633358306354,
  0.16187661325946062,
  0.3006715892920996,
  0.42809462336187576,
  0.5790540149106302,
  0.706221702530327,
  0.8352987384060198,
  0.9707840531576964,
  1.0963631997951027
]
[
  0.020927754394140415,
  1.277633358306354,
  0.16187661325946062,
  0.3006715892920996,
  0.42809462336187576,
  0.5790540149106302,
  0.706221702530327,
  0.8352987384060198,
  0.9707840531576964,
  1.0963631997951027
]
....
....
```
--
```
cat fp8_fp8_tp2.json | jq '.benchmarks[] | .metrics.requests_per_second.successful.mean'

0.020927754394140415
1.277633358306354
0.16187661325946062
0.3006715892920996
0.42809462336187576
0.5790540149106302
0.706221702530327
0.8352987384060198
0.9707840531576964
1.0963631997951027
```

Looks like an unintentional referencing of the same object between benchmark runs ?

Action Item: 
Strip this out from the output file. 



Closing this out as the output formats have been changed and cleaned up significantly
Repository: vllm-project/guidellm
Issue: Add Docker and Kubernetes Support for GuideLLM Benchmarking
Root cause: Unimplemented Feature Gap

We need a dockerfile to enable running GuideLLM benchmarks in a containerized environment and as a Kubernetes job. This would be helpful for users who want to run benchmarks in a more controlled and scalable environment.

The first change should include:

**Dockerfile**
- Based on Python 3.11 slim image
- Installs GuideLLM and its dependencies
- Entrypoint to run benchmarks with environment variables
Repository: vllm-project/guidellm
Issue: Discrepancy between max-requests parameter and actual number of unique requests generated
Root cause: Documentation Deficiency

When running the benchmark with the --max-requests parameter set to 100 and using a data parameter in the format `prompt_tokens=256,output_tokens=128`, the request loader creates 1000 unique requests instead of the expected 100.

**Steps to reproduce:**
1. Run the benchmark with the following command:
`guidellm benchmark --target "http://localhost:8000" --model "meta-llama/Llama-3.1-8B-Instruct" --rate-type "sweep" --data "prompt_tokens=256,output_tokens=128" --max-requests 100 --max-seconds 100 --output-path "/mnt/requests/guidellm_results.json"`

Expected behavior:
- The request loader should create exactly 100 unique requests as specified by the --max-requests parameter.
Actual behavior:
- The request loader creates 1000 unique requests, as shown in the log message:

`Created loader with 1000 unique requests from prompt_tokens=256,output_tokens=128.`

Root cause:
The issue appears to be in the `GenerativeRequestLoader` class, where it's generating a fixed number of unique requests (1000) when the data parameter is specified as token parameters, rather than respecting the max_requests parameter.
Impact:
This behavior can lead to:
- Inconsistent benchmarking results
- Longer than expected benchmark runs
- Potential resource wastage when the actual number of requests needed is much smaller

`Created loader with 1000 unique requests from prompt_tokens=256,output_tokens=128.`

Is indicating that the synthetic data generator created a dataset with 1000 rows

When you set `--max-requests 100` only 100 of those rows will be utilized in the benchmark (x10 since `--max-requests` and `--max-seconds` apply to each rps/concurrency level separately).

Hi @wangchen615, just checking in here if you are still seeing an issue here. As @sjmonson noted, the printout on the console is stating that it created 1000 unique requests to run for the benchmarks -- it's set to this number to ensure there is enough variety to prevent impacts from prefix caching and things like that whether it's within a single benchmark or across multiple on the server side. You should still see each benchmark running for the desired number of requests or less if max-seconds is hit first. You can check this in the progress output for completed requests, the table output at the end in the console for the number of requests, as well as in the output file.
Repository: vllm-project/guidellm
Issue: support for multi-turn qa dataset
Root cause: Unimplemented Feature Gap

Hi Team,
I was wondering if we can add support for the following multi-turn datasets 
1) https://github.com/LMCache/LMCache/tree/dev/benchmarks/multi-round-qa
2) https://github.com/ekwinox117/multi-challenge 
3) https://huggingface.co/datasets/bryanlincoln/infoquest

so that we don't have to run their benchmarking scripts for these specific datasets that are becoming quite important to evaluate cost/accuracy tradeoffs in agentic workflows.

cc @sjmonson @dagrayvid @markurtz 

Will be implemented / fixed by #374
Repository: vllm-project/guidellm
Issue: broken link in the docs

The link 
"[Outputs documentation](https://github.com/neuralmagic/guidellm/blob/main/raw.githubusercontent.com/neuralmagic/guidellm/main/docs/outputs.md)
at
https://github.com/neuralmagic/guidellm#4-explore-the-results-file

is broken
Repository: vllm-project/guidellm
Issue: Event loop is closed error.
Root cause: Resource Mishandling

I’m encountering the following error for some requests when running the GuideLLM benchmark command using version v0.2.1 with Python 3.12.9.

The command I’m using is:

```
guidellm benchmark \
  --target "<valid target>" \
  --model "<model>" \
  --rate-type constant \
  --rate 0.31 \
  --data "prompt_tokens=4096,output_tokens=4096" \
  --processor "<valid tokenizer>" \
  --max-seconds 300 \
  --max-requests 100 \
  --output-path test.csv
```

And the error I’m seeing intermittently is:

` text_completions | ERROR - OpenAIHTTPBackend request with headers: {'Content-Type': 'application/json', 'Authorization': 'Bearer <valid token>'} and payload: {'prompt': '; <valid prompt>', 'model': '<valide model>', 'stream': True, 'stream_options': {'include_usage': True}, 'max_tokens': 128, 'max_completion_tokens': 128, 'stop': None, 'ignore_eos': True} failed: Event loop is closed`

Has anyone else encountered this issue? If so, was there a resolution or workaround that worked for you?

I saw the same error with the configuration below on multiple endpoints, on being a L40s with vllm 0.8.2 that has been reliable for us, so I suspect its on the testing side.
guidellm benchmark \
     --target "" \
     --model "" \
     --processor"" \
       --rate-type sweep \
        --max-seconds 30 \
        --max-requests 128 \
        --data "prompt_tokens=512,output_tokens=512"
Repository: vllm-project/guidellm
Issue: Support VL Model Benchmarking
Root cause: Unimplemented Feature Gap

Description: 
Add support into GuideLLM to benchmark VL Models (Images in, Text out). 

**Acceptance Critieria**

This would be a great new feature to add, also want to ask if we can support benchmarking with multiple images per request, not necessarily just one image.
Repository: vllm-project/guidellm
Issue: Sweep Mode Improvements - get closer to approaching peak-throughput
Root cause: Algorithmic Error

Description: 
Improve sweep mode to find a way to get an amount over throughput to match what we would get for a peak. 

**Acceptance Criteria:**

Closing out with the following finding shared internally:

Fun finding towards wether or not our throughput measurement is accurate or not (we've had some discussions on if it would overload the server too much and underrepresent the true throughput limit) based on some experiments we're doing on the Neural Magic research team side to model speculative decoding performance. These were the results from both doing a halving of the times towards the limit in sweep rather than linear, but additionally it was adding 10% onto the throughput measurement as a safety to ensure we measured the limit correctly (really important for comparison to the analytical model). Within the underlying data, there's a clear limit where as soon as it exceeds the measured throughput, it starts over saturating and creating that cluster of dots towards the middle of the graph.

The dot in the upper right corner is throughput, the dot in the lower left corner is synchronous, everything else is constant rate

<img width="579" height="455" alt="Image" src="https://github.com/user-attachments/assets/8b1d1e2f-0aa6-4e9c-a55d-002920939c5a" />
Repository: vllm-project/guidellm
Issue: Python 3.9 does not support `int.is_integer()`
Root cause: Environment Incompatibility

Versions of python older than 3.12 do not support `int.is_integer()` which can happen when `--rate` is called with a whole number:

```python
Traceback (most recent call last):
  File "/opt/app-root/bin/guidellm", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/src/guidellm/src/guidellm/__main__.py", line 255, in benchmark
    asyncio.run(
  File "/usr/lib64/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/opt/app-root/src/guidellm/src/guidellm/benchmark/entrypoints.py", line 88, in benchmark_generative_text
    profile = create_profile(rate_type=rate_type, rate=rate)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/src/guidellm/src/guidellm/benchmark/profile.py", line 380, in create_profile
    return ConcurrentProfile.from_standard_args(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/src/guidellm/src/guidellm/benchmark/profile.py", line 138, in from_standard_args
    if not all(stream.is_integer() and stream > 0 for stream in rate):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/src/guidellm/src/guidellm/benchmark/profile.py", line 138, in <genexpr>
    if not all(stream.is_integer() and stream > 0 for stream in rate):
               ^^^^^^^^^^^^^^^^^
AttributeError: 'int' object has no attribute 'is_integer'
```
Repository: vllm-project/guidellm
Issue: [Feature Request] Record server config from /server_info endpoint when vLLM 0.9.0 lands
Root cause: Unimplemented Feature Gap

If we can record the server config as part of the benchmark data via this endpoint, this would make life a whole lot easier to compare across runs. 
The feature implementation for this would also require to check for the availability for the endpoint considering this is an optional endpoint.

Reference to the vLLM issue : https://github.com/vllm-project/vllm/pull/16572
Repository: vllm-project/guidellm
Issue: Re-load and re-display a saved benchmark.json report
Root cause: Unimplemented Feature Gap

**Description:**
Enable the ability to load up and display a previous benchmark.json report in the terminal. 

**Acceptance Criteria:**
- Enable a `--view-report benchmark.json` command to surface an existing benchmark.json file result in the CLI (and in the future the UI)
Repository: vllm-project/guidellm
Issue: Automatically report stabilized performance - early stopping when peak throughput is reached
Root cause: Unimplemented Feature Gap

**Description:**
Enable early stopping in a GuideLLM benchmark when the peak throughput has been reached. As soon as the max throughput is reached and there is no longer more room (we can deem this as X times of +- 2% improvement), stop the benchmark early as it is just wasting GPU resources at that point. 

**Acceptance Criteria:**
To be defined

I think this should also be accompanied by a minimum duration flag/control, an env variable should be perfectly okay. There might be scenarios where want to give the servings some time to start responding at the right throughput range.
Repository: vllm-project/guidellm
Issue: Add support for a Python vLLM backend
Root cause: Unimplemented Feature Gap

**Description:**
Right now, GuideLLM only works w/ GuideLLM through the OpenAI compatible API backend to take the target URI and perform benchmarking on a vLLM server. We want to also add support for the python vLLM backend so that we can create scripts for benchmarks w/ vLLM and use it in-line with Python. 

**Acceptance Criteria:**
- Add support for the Python vLLM Backend
- Potentially helpful: https://github.com/triton-inference-server/vllm_backend - maybe not
Repository: vllm-project/guidellm
Issue: Token Efficiency Benchmarking
Root cause: Unimplemented Feature Gap

Benchmarks that crown the “best” reasoning LLMs routinely publish a single headline score, yet ignore how many tokens—and therefore how much time, energy, and money—the model spent to achieve that score. When practitioners must meet strict service‑level objectives (SLOs) for latency and cost, a system that needs five‑times more tokens per answer is rarely competitive, even if its accuracy is marginally higher. Recent work on agent evaluation has highlighted similar blind spots by arguing for cost‑controlled leaderboards and Pareto visualizations of accuracy versus dollars. Building on this insight, we must shift the focus from dollars to the more fundamental currency of inference: tokens.

Token Efficiency Benchmarking (TEB) will be a systematic framework that measures the quality–token trade‑off of reasoning LLMs and test‑time compute optimization techniques. For each model or technique, TEB will record 
(i) task‑level quality metrics—e.g., exact‑string accuracy on GSM‑8K or pass@k on HumanEval—and 
(ii) the total number of prompt, generation, and tool‑use tokens consumed to reach the reported score. 

The ask is for implementing TEB in guidellm.

cc @rgreenberg1 @sjmonson @markurtz @dagrayvid
Repository: vllm-project/guidellm
Issue: Ensure prefixes cannot be cached for synthetic datasets
Root cause: Unimplemented Feature Gap

For the SyntheticTextItemsGenerator, we need to add a controllably different prefix or start to each request to ensure automatic prefix caching within vLLM is never hit. Specifically for the create_prompt pathway, we need to set the start text as an auto-incrementing number, for example, for the requested number of items.

The validating test is that for each request made out of the total requests generated, the string match starting from the first character will always fail across requests and will not be equal. The end to end test is running a benchmark with the latest vLLM version and checking the prefix caching hit rate, which should evaluate to 0%
Repository: vllm-project/guidellm
Issue: Improve performance of synthetic dataset prompt creation to match a given number of tokens
Root cause: Algorithmic Error

Currently, the synthetic dataset prompt creation process is using a binary search to converge on the desired prompt length based on the input prompt. This means that a large number of requests are being sent to the relatively expensive tokenize function for the processor. Recently, #162 was added, which enabled a significantly cheaper way to generate a prompt of a given length. This involves grabbing a prompt of a given length, tokenizing it, truncating the tokens array to the desired length, and reencoding it back to text. Provided we have a reasonable start multiplier (likely 3 or 4 tokens per word, which should be double-checked against the average for current tokenizers, or we dynamically calculate the tokens per word ratio as prompts are generated), then most will result in only a single tokenization call. If we add a safety that multiplies by a reasonable constant for the number of words to target if it doesn't meet the token length constraints after tokenizing, then we can guarantee proper convergence.

The test for this will be to ensure that the ratio of tokenize calls to prompts is approximately one and the average truncated number of tokens relative to the desired number of tokens is reasonably small (20%) while the number of tokens after running a benchmark through vLLM matches the number of prompt tokens desiredCurrently, our method for creating synthetic dataset prompts uses a binary search to establish the desired prompt length, leading to numerous requests to the expensive tokenization function.
Repository: vllm-project/guidellm
Issue: Long running test spawn too many threads
Root cause: Resource Mishandling

The `SyncManager` process spawns a huge number of threads over the lifetime of the benchmark. On pid constrained systems (such as OpenShift) we can run out of threads during longer tests or with shorter requests.

This issue still occurs after replacing the `SyncManager` with shared queues.

After a bunch of testing I've found that the issue only occurs with Python 3.12.

Need to double check this after #251 lands

Should be fixed now with 0.3
Repository: vllm-project/guidellm
Issue: Add new benchmark mode to search for peak goodput under an SLO
Root cause: Unimplemented Feature Gap

Often when we benchmark a new model or hardware, the goal is to determine the max RPS or tokens per second that the server can sustain under a certain SLO. We should add a new feature similar to the "sweep" but instead of doing linearly spaced constant RPS runs, it should do something like a binary search to try to find the peak load which the server can handle while meeting a defined latency SLO. 

We would need to support some config options for the SLO, to support p99 or p95 ITL and TTFT. 

I have a rough PoC of this in progress on this branch: https://github.com/dagrayvid/guidellm/tree/goodput, but wanted to open this issue to discuss the idea further and track progress.
Repository: vllm-project/guidellm
Issue: Simplifying the readme towards expected use cases and examples; expand out docs particularly for configuration options
Root cause: Documentation Deficiency


@markurtz can i apply for this task? but i don't know the exactly meaning, could you give some example about what use cases or configuration options?

Hi @tukwila, yes, I'll gather some requirements on this and send over later. We have a large refactor in place for 0.4 which will need a lot of docs updated, and where ever you can help within that, that would be great!

> Hi [@tukwila](https://github.com/tukwila), yes, I'll gather some requirements on this and send over later. We have a large refactor in place for 0.4 which will need a lot of docs updated, and where ever you can help within that, that would be great!

Yes, i'd like to do so; and if any other modules/functions refactor, i'd like to provide my effort.
Repository: vllm-project/guidellm
Issue: Build official container image containing guidellm
Root cause: Documentation Deficiency

It would be great to have an official container image with guidellm pre-packaged that can be easily consumed for running guidellm from inside a container.

As a reference, I have started a repo to help build an "unofficial" container here:

https://github.com/redhat-ai-services/guidellm-helper

Which publishes an image here:

https://quay.io/repository/redhat-ai-services/guidellm?tab=tags

I'm happy to help contribute it back to this repo if there is interest.


Hi, we do have official container images published at `ghcr.io/neuralmagic/guidellm` with pending CI automation (see #190). If you want to contribute any improvements to our [official Containerfile](https://github.com/neuralmagic/guidellm/blob/main/deploy/Containerfile) feel free.