zhiminy/EvalEng huggingface dataset:
Repository: vllm-project/guidellm
Issue: Guidellm adds unexpected field to requests
Root cause: Interface Contract Mismatch

**Describe the bug**
Guidellm adds the `max_completion_tokens` field to requests. As a result we see this log on vllm: 

```bash
WARNING 08-09 20:50:05 [protocol.py:58] The following fields were present in the request but ignored: {'max_completion_tokens'}
WARNING 08-09 20:50:05 [protocol.py:58] The following fields were present in the request but ignored: {'max_completion_tokens'}
WARNING 08-09 20:50:05 [protocol.py:58] The following fields were present in the request but ignored: {'max_completion_tokens'}
WARNING 08-09 20:50:05 [protocol.py:58] The following fields were present in the request but ignored: {'max_completion_tokens'}
WARNING 08-09 20:50:05 [protocol.py:58] The following fields were present in the request but ignored: {'max_completion_tokens'}
WARNING 08-09 20:50:05 [protocol.py:58] The following fields were present in the request but ignored: {'max_completion_tokens'}
WARNING 08-09 20:50:05 [protocol.py:58] The following fields were present in the request but ignored: {'max_completion_tokens'}
WARNING 08-09 20:50:05 [protocol.py:58] The following fields were present in the request but ignored: {'max_completion_tokens'}
```


**Expected behavior**
- it should not add this feild

**Environment**
Include all relevant environment information:
1. OS [e.g. Ubuntu 20.04]:
2. Python version [e.g. 3.12.2]:

**To Reproduce**
Exact steps to reproduce the behavior:


**Errors**
If applicable, add a full print-out of any errors or exceptions that are raised or include screenshots to help explain your problem.

**Additional context**
Add any other context about the problem here. Also include any relevant files.


We currently keep `max_completion_tokens` for compatibility (See #210). In the future we may support custom request templates and default to only having max_tokens #194).

v0.4.0 will now have custom request templates to support this
Repository: vllm-project/guidellm
Issue: Guidellm takes several minutes to create random requests with long prompts
Root cause: Algorithmic Error

**Describe the bug**
It takes several minutes to generate random requests

```bash
guidellm benchmark --target=$URL --model=$MODEL --rate-type=concurrent --rate=200 --max-requests=200 --output-path=~/llama-70b.json --processo
r=$MODEL --data='{"prompt_tokens":20000, "output_tokens":5000}'
```

This takes multiple minutes at:

```bash
Creating backend...
Backend openai_http connected to http://10.16.1.185:8000 for model meta-llama/Llama-3.3-70B-Instruct.
Creating request loader...
```

**Expected behavior**
- I would expect this to take a shorter amount of time

**Environment**
Include all relevant environment information:
1. OS [e.g. Ubuntu 20.04]:
2. Python version [e.g. 3.12.2]:

**To Reproduce**
Exact steps to reproduce the behavior:


**Errors**
If applicable, add a full print-out of any errors or exceptions that are raised or include screenshots to help explain your problem.

**Additional context**
Add any other context about the problem here. Also include any relevant files.


https://github.com/vllm-project/guidellm/pull/320
This might not be the root cause, but it can help mitigate the problem

Noting in here, this will be fixed in 0.4 now with a better algorithm for determining the proper number of tokens when generating synthetic

0.4.0 has landed and has a fix in there for this in addition to defaulting the request generation to be on demand rather than precreating
Repository: vllm-project/guidellm
Issue: `--rate-type=sweep` does not load vllm with expected count of requests
Root cause: Algorithmic Error

**Describe the bug**
I ran sweep test to understand the maximum throughput of the service. guidellm calculated 0.5 RPS (though `vllm/benchmarks/benchmark_serving.py ` showed 0.7 RPS when iterating through concurrency).
I logged into the vllm graphana to observe the load on the service. guidellm says 0.5-0.55 RPS. But in fact, I see 13-15 requests.

**Expected behavior**
0.5 RPS = 30 RPM. Shouldn't 30 requests be processed if the service holds 0.5 RPS?

**Environment**
Include all relevant environment information:
1. OS [e.g. Ubuntu 20.04]: -
2. Python version [e.g. 3.12.2]: 3.10
3. Docker-image: python:3.10-slim

**To Reproduce**
Exact steps to reproduce the behavior:
```
> pip install -U git+https://github.com/vllm-project/guidellm.git@1261fe81c57b07ed64333b5d50846699aa5307d4
> export GUIDELLM__PREFERRED_ROUTE="chat_completions" && export GUIDELLM__OPENAI__MAX_OUTPUT_TOKENS=512 && export GUIDELLM_MAX_REQUESTS=1000 && export GUIDELLM__REQUEST_TIMEOUT=600 && guidellm benchmark --target http://localhost:8000 --rate-type sweep --model Qwen/Qwen3-30B-A3B --processor Qwen/Qwen3-30B-A3B --random-seed 2025 --max-seconds 300 --data "prompt_tokens=4096,output_tokens=512,samples=1000" --backend-args '{"extra_body":{"chat_template_kwargs":{"enable_thinking":false}}}' --output-path "data/benchmarks.json"
```

**Errors**
guidellm:
<img width="1113" height="404" alt="Image" src="https://github.com/user-attachments/assets/f42f9f17-ca5a-4dc7-b598-d82e44ead44d" />

vllm grafana:
<img width="940" height="391" alt="Image" src="https://github.com/user-attachments/assets/f4a9e7b2-136a-4b2c-aa99-8399ad5b3f00" />

**Additional context**
Add any other context about the problem here. Also include any relevant files.
Repository: vllm-project/guidellm
Issue: UI erroneously labels ITL as TPOT
Root cause: Validation Gap

**Describe the bug**
Graphs in the benchmark UI show TPOT in places which are actually showing ITL.

**Expected behavior**
All graphs that are labeled with "TPOT" should instead be labeled with "ITL".

Fix here: https://github.com/vllm-project/guidellm/pull/288
Opted to switch the data from ITL to TPOT since it is the simplest fix and then we don't have to use ITL when we've always had TPOT in the UI. If ITL is preferred though I can make that change

New PR up for the fix: #386
Repository: vllm-project/guidellm
Issue: UI RPS label unclear
Root cause: Documentation Deficiency

**Describe the bug**
In the summary graph of the UI. The X-axis is labeled as "Maximum RPS Per GPU". Since we cannot measure how may GPUs the server is using this is probably inaccurate.

**Expected behavior**
Label should be something like "Measured RPS (Mean)" or "Responses Per Second (mean)".


Fix: https://github.com/vllm-project/guidellm/pull/298

#298 was merged in, this is fixed!
Repository: vllm-project/guidellm
Issue: Inconstant use of abbreviations in UI
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
The summary graph in the UI uses a mix of abbreviation and full names for the y-axis labels. For the sake of new users it would be best to convert all of these labels to their full metric names.

**Describe the solution you'd like**
Replace TTFT -> Time Per Output Token and ITL -> Inter-Token Latency.


#388 Fixes this

I think we can close this @sjmonson
Repository: vllm-project/guidellm
Issue: Label points in UI with what benckmark they represent
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
The summary graph in the UI plots each benchmarks Responses Per Second against various latency metrics and interpolates values between the points. Because there are no points on the graph it is hard to understand where the measured values are.

**Describe the solution you'd like**
Add dots at each measured value and a tooltip/label matching the benchmark to a row in the CLI summary. (e.g. `constant@0.5`)

#390 fixes this

Update, now #392 fixes this and will likely be made available in the 0.4.0 release coming soon.
Repository: vllm-project/guidellm
Issue: 评测自己的服务发生报错AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
Root cause: Resource Mishandling

**Describe the bug**
当我使用本项目评测我本地启动的模型服务时，跑一会就会发生报错

<img width="1250" height="570" alt="Image" src="https://github.com/user-attachments/assets/d8c83cf8-e95b-4d27-879a-9ecd597c2b2f" />

**start command**
python run.py benchmark --target "http://10.16.2.3:11092" --model Qwen2_5-VL-7B-Instruct --processor /Users/huizhang/Develop/ai/model/Qwen2.5-VL-7B-Instruct --rate-type sweep --max-seconds 60 --data "prompt_tokens=512,output_tokens=10"

**Environment**
Include all relevant environment information:
1. OS: macOS 12.7.3
2. Python version : 3.10.15

帮忙确定一下是什么环节的问题


This may be fixed with #251 and 0.3.0. Once that goes out, will reping to double check and see if we need to schedule any follow up work

OK. Thanks

@Ladazhx  please double check if this is now working with the 0.3.0 that went out. If not, 0.4.0 will enable full control over how the processes are forked/created

Closing out as 0.4.0 should have fixed this, please reopen if not
Repository: vllm-project/guidellm
Issue: Unable to test gemini service
Root cause: Architectural Constraint

**Describe the bug**
gemini does not support endpoints with `/v1/...`, and there is no way to override it.

**Expected behavior**
is it possible to add the ability to redefine endpoints for certain services/models?

**Environment**
Include all relevant environment information:
1. OS [e.g. Ubuntu 20.04]: -
2. Python version [e.g. 3.12.2]: 3.10

**To Reproduce**
I've tried using the gemini documentation for load testing via the guideline: https://ai.google.dev/gemini-api/docs/openai
```bash
export GUIDELLM__PREFERRED_ROUTE="chat_completions" &&
      export GUIDELLM__OPENAI__MAX_OUTPUT_TOKENS=512 &&
      export GUIDELLM__MAX_CONCURRENCY=233 &&
      export GUIDELLM__REQUEST_TIMEOUT=300 &&
      export GUIDELLM__PREFERRED_PROMPT_TOKENS_SOURCE=local &&
      export GUIDELLM__PREFERRED_OUTPUT_TOKENS_SOURCE=local &&
      guidellm benchmark
      --target https://generativelanguage.googleapis.com/v1beta/openai/
      --rate-type sweep --rate 5
      --model gemini-2.5-flash --processor ${processor}
      --random-seed 1234
      --max-requests=100
      --data "prompt_tokens=4096,output_tokens=512"
      --backend-args '{"extra_body":{"chat_template_kwargs":{"enable_thinking":false}}, "headers":{"Authorization": "Bearer GEMINI_API_KEY"}}'
      --output-path "${output_dir}/benchmarks.json"
```

**Errors**
The error is that guidellm could not get a list of available models by endpoint `/v1/models`

**Additional context**
Add any other context about the problem here. Also include any relevant files.


@sjmonson 
I wrote adapter to gemini to test the service through guidellm. But I noticed that gemini/openai-gpt do not support ignore_eos. 
Can you tell me if there is any way to configure the generation of prompta in guidellm so that there is some kind of instruction at the beginning of the text (for example, do not generate eos, as in llmperf)?

If the model server does not support `ignore_eos` or an equivalent like `min_tokens` then there is no way to get an exact number of tokens from the server in a single request. I have never heard of `llmperf` but a model cannot just decide to not send eos tokens based on user prompting.

Noting here that the 0.4 release will have fully customizable pathways for the request formats that are sent. The default will still run as is, but you'll have workarounds with a bit of work. Adding to 0.4 milestone for when that is released

@markurtz  was this picked up in v0.4.0 or will it be taken in a future release ?

Yes, this did land in v0.4.0. You can adjust routes with something like: `--backend-kwargs '{"api_routes":{"health": "health", "models": "v1/models", "chat_completions": "v1/chat/completions"}}'`. You can see the full list here:

https://github.com/vllm-project/guidellm/blob/765c875a7378d916fe22accd6b66371062a43d54/src/guidellm/backends/openai.py#L83-L90

At minimum you need to set `health` and whichever completions endpoint you are using. `models` is necessary if you do not specify the model name.
Repository: vllm-project/guidellm
Issue: Create a contrib folder for user submitted and maintained guidellm output visualization
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
1. It is difficult to standardize on a specific visualization for guidellm collected metrics as users have different preferences.
2. At the same time, many users may not want to write their if what they are looking is already available in the repo

**Describe the solution you'd like**
Guidellm ships with its official HTML visualization. Create a contrib section in the repo to accept user submitted visualizations with appropriate CI tests. 

**Describe alternatives you've considered**
Alternative is for users to maintain the visualizations in their own repos but they might be broken as guidellm evolves and it is hard to share

**Additional context**
Example user created visualization

[concurrency-sweep-test.html](https://github.com/user-attachments/files/22181179/concurrency-sweep-test.html)

Adding this into the 0.5 so we can scope out what a more extensive solution would look like both for examples as well as contributed pathways for the new refactor plugins
Repository: vllm-project/guidellm
Issue: Synthetic data generator creates excessive samples when --max-requests is set
Root cause: Documentation Deficiency

**Describe the bug**
When running a benchmark with synthetic data, the number of samples generated defaults to 1000. If a user specifies a small number of total requests (e.g., --max-requests=10), the request loader still generates the full 1000 samples. This results in a very long and unexpected delay at the "Creating request loader..." step, even for small benchmark runs.

**Expected behavior**

  When using synthetic data, if the samples parameter is not explicitly set in the --data configuration, the number of samples generated should 
  intelligently default to the value of --max-requests if it is provided.

  For example, if a user runs a benchmark with --max-requests=10, the request loader should only generate 10 synthetic samples, not 1000. This would 
  make the startup time for small tests significantly faster and the tool's behavior more intuitive.

**Environment**
Include all relevant environment information:
1. OS [e.g. Ubuntu 20.04]:
2. Python version [e.g. 3.12.2]:

**To Reproduce**
guidellm benchmark run --max-requests 50 --data='{"prompt_tokens":100, "output_tokens":100}'

Before this PR: Observe the long delay and the log message: Created loader with 1000 unique requests...


**Errors**
If applicable, add a full print-out of any errors or exceptions that are raised or include screenshots to help explain your problem.

**Additional context**
Add any other context about the problem here. Also include any relevant files.


We want to be able to control the dataset length independently from `max-requests` for a couple reasons. Firstly constants (`max-seconds`, `max-requests`, etc) are supposed to only be used as a stop condition and not affect any other test parameters. Secondly oversampling allows each test in a series (such as sweep) to use a different set of samples.

The intended mechanism for configuring sample count is the `samples` arg to the synthetic dataset generator. E.g. `--data='{"prompt_tokens":100, "output_tokens":100, "samples":10}'`.

We also will have some changes in the future that will hopefully speed up sample generation as our current algorithm is quite inefficient; especially for large prompts.

Adding on to this to retest after 0.4 goes out because this will have performance improvements for synthetic data generation
Repository: vllm-project/guidellm
Issue: Redundant backend validation in concurrent worker processes
Root cause: Architectural Constraint

**Describe the bug**

When a concurrent benchmark is initiated (e.g., with --rate-type=concurrent), the main process correctly creates and validates the backend. 
However, when the worker processes are subsequently spawned for the test, each worker process calls the backend.validate() method again.

This results in multiple, redundant "Test connection" network requests being sent to the target endpoint: one for the main process, and one for each worker process.

**Expected behavior**

The backend validation should only be performed once by the main process before the workers are created. Worker processes, which receive a copy of the already-validated backend object, should not perform this validation again.

This would prevent unnecessary network requests, reduce benchmark startup latency, and make the logs cleaner.

**Environment**
Include all relevant environment information:
1. OS [e.g. Ubuntu 20.04]:
2. Python version [e.g. 3.12.2]:

**To Reproduce**

guidellm benchmark run \
         --target "http://your-api-endpoint/v1" \
         --rate-type=concurrent \
         --rate=2 \
          --max-requests 2 \
          --data='{"prompt_tokens":32,"output_tokens":32}'

Observe the console output. You will see the log message validating backend and the associated "Test connection" request logs appear multiple
      times (once for the main process, and once for each of the 2 worker processes).

**Errors**
If applicable, add a full print-out of any errors or exceptions that are raised or include screenshots to help explain your problem.

**Additional context**
Add any other context about the problem here. Also include any relevant files.


<img width="2356" height="492" alt="Image" src="https://github.com/user-attachments/assets/e10a4c0c-5eef-4b1d-af98-b59ec1e11849" />
Repository: vllm-project/guidellm
Issue: Proposal: Change default route to /v1/chat/completions
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

"I propose setting /v1/chat/completions as the default route. This would align the tool with current industry best practices, where the chat 
  completions endpoint is the definitive standard. It streamlines the setup for the vast majority of users, while ensuring backward compatibility is 
  maintained for the minority who rely on the legacy /v1/completions endpoint, as they can opt-in to the old behavior by setting 
  GUIDELLM__PREFERRED_ROUTE="text_completions"."

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.


Won't Do. `chat/completions` adds extra tokens to the prompt which bias testing. Additionally we rely on tunables which are not officially part of the `chat/completions` spec. We only support `chat/completions` for when the text `completions` is unavailable.
Repository: vllm-project/guidellm
Issue: [Refactor] guidellm.benchmark package rewrite
Root cause: Unimplemented Feature Gap

Updates and refactors for the GuideLLM platform rewrite within the guidellm.benchmark package including improvements to the benchmarker and entrypoints logic for better, more performant parallelization and the additions of aggregators, outputs, and profile expansions for a more pluggable framework
Repository: vllm-project/guidellm
Issue: [Refactor] guidellm.data package rewrite and tests
Root cause: Unimplemented Feature Gap

Updates and refactors for the GuideLLM platform rewrite within the guidellm.data related packages including improvements in pluggability with new concepts of deserializers, formatters, preprocessors, loaders, and datasets in addition to expansion of capabilities with full multi-modal support
Repository: vllm-project/guidellm
Issue: [Refactor] implement the guidellm.mock_server package
Root cause: Unimplemented Feature Gap

Add a new mock_server for testing and performance validations of the GuideLLM system with a Sanic server that mocks the common OpenAI/vLLM routes

@markurtz  Can i apply for this?

@markurtz please review: https://github.com/vllm-project/guidellm/pull/380

Closing out as completed
Repository: vllm-project/guidellm
Issue: [Refactor] multi-turn support built on top of the new rewrite
Root cause: Unimplemented Feature Gap


I've been testing multi-turn benchmarking against vllm and llm-d.  Here is a repo I've put together with a sample app:  https://github.com/rh-aiservices-bu/multi-turn-benchmarking

I'd be interested in helping incorporate this type of functionality to guidellm.  It's essential to reliably demonstrate the improvements provided by intelligent scheduling from llm-d.
Repository: vllm-project/guidellm
Issue: [Refactor] guidellm.benchmark shared pathway
Root cause: Architectural Constraint

We would benefit from a shared pathway for both the `benchmark run` and `benchmark from-file` pathways due to the new logic for outputs. Otherwise it won't support multiple output types, or the logic for selecting the output types in the new setup.

While doing so I would prefer to split up the `benchmark_generative_test` function since it is absolutely gigantic. I plan to reuse some of the code from that in `from-file`.

Closing this out as completed with 0.4
Repository: vllm-project/guidellm
Issue: start_tokens in synthetic prompt is not related to prompt_text
Root cause: Algorithmic Error

**Describe the bug**
A clear and concise description of what the bug is.

When i do benchmark testing using synthetic prompts, the prompts are started with not unified language such as:

In this screenshot, start_tokens are different languages that are followed by english.
<img width="1125" height="410" alt="Image" src="https://github.com/user-attachments/assets/371e5ec0-9169-46fe-b4e6-22934ca341c8" />


In this screenshot, start_tokens are different with following German.
<img width="1344" height="400" alt="Image" src="https://github.com/user-attachments/assets/c1ac4ddd-bb82-4e9c-8ef1-e71d538120fa" />


Such prompt_tokens are inappropriate for benchmark testing.

and i write one script to dig into this problem:

```
import json

from guidellm.dataset.synthetic import (
    SyntheticDatasetCreator,
)

def test_handle_create_basic():
    # Example: German
    # data = "prompt_tokens=128,output_tokens=56, samples=10, source=https://www.gutenberg.org/cache/epub/30793/pg30793.txt" 
    # Example: English
    data = "prompt_tokens=128,output_tokens=56, samples=10, source=https://www.gutenberg.org/cache/epub/1342/pg1342.txt" 
    synthetic_creator=SyntheticDatasetCreator()
    res_list = []
    result = synthetic_creator.handle_create(
            data=data,
            data_args=None,
            processor="${local_path}/Qwen2.5-1.5B-Instruct",
            processor_args=None,
            random_seed=42,
        )
    res_list = result[:]['prompt']
    with open('./syntheic_prompts.json', 'w', encoding='utf-8') as f:
        json.dump(res_list, f, ensure_ascii=False, indent=4)
        
if __name__ == "__main__":
    test_handle_create_basic()
```

**Expected behavior**
A clear and concise description of what you expected to happen.

start word should be the same with the whole prompt.

**Environment**
Include all relevant environment information:
1. OS [e.g. Ubuntu 20.04]: 
2. Python version [e.g. 3.12.2]: Python 3.9.9
3. guidellm version: 0.3.0

**To Reproduce**
Exact steps to reproduce the behavior:
step1. copy self-test scripts from above into local env.
step2. python ./test_synthetic_prompt.py

**Errors**
If applicable, add a full print-out of any errors or exceptions that are raised or include screenshots to help explain your problem.

**Additional context**
Add any other context about the problem here. Also include any relevant files.
Repository: vllm-project/guidellm
Issue: Make base server endpoint configurable
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Sometimes OpenAI-compatible endpoints are hosted on differing paths from `/v1/...` (e.g. `/v3/...`, `/v1beta/...`)

**Describe the solution you'd like**
Support a config option for configuring the base endpoint. Something like `GUIDELLM__OPENAI__BASE_ENDPOINT=/v1`.

This is now supported through the backend endpoints configuration for the OpenAIBackend