zhiminy/EvalEng huggingface dataset:
Repository: vllm-project/guidellm
Issue: Possible invalid request formatting for `max_completion_tokens`
Root cause: Configuration Error

Just looking in to this, but wanted to report it in case it was a known issue or someone has more information.

While running guidellm against a locally running `vllm serve`, I am seeing a very large amount of these log messages in the vLLM output:

```txt
WARNING 06-27 14:57:36 [protocol.py:58] The following fields were present in the request but ignored: {'max_completion_tokens'}
```

Running a request manually against the endpoint is happy, with no errors in the vllm logs:

```jsh
curl -k -s -H "Content-Type: application/json" http://localhost:8000/v1/chat/completions -d '{"model":"llama3.1-8b-instruct","messages":[{"role":"user","content":"What is an AI tensorized weight?"}],"max_completion_tokens":35}' | jq .
```

```json
{
  "id": "chatcmpl-e00ce83f-121f-482a-b43c-5c4494ad29ae",
  "object": "chat.completion",
  "created": 1751037186,
  "model": "llama3.1-8b-instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "reasoning_content": null,
        "content": "A tensorized weight, also known as a tensor weight or a weight tensor, is a type of weight used in artificial neural networks (ANNs) and deep learning models.",
        "tool_calls": []
      },
      "logprobs": null,
      "finish_reason": "length",
      "stop_reason": null
    }
  ],
  "usage": {
    "prompt_tokens": 43,
    "total_tokens": 78,
    "completion_tokens": 35,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null
}
```

Which leads me to believe the prompt being formed by guidellm must be placing `max_completion_tokens` somewhere other than as a top level property of the request struct.

Tldr: you can ignore that error. We set both `max_completion_tokens` and `max_tokens` to the same value.

The standard for setting a max tokens is kinda of messy. Some model servers support `max_tokens` as the flag and some support `max_completion_tokens`. In the official OpenAI documentation the legacy endpoint only supports `max_tokens` while the chat endpoint supports `max_tokens` for older models and `max_completion_tokens` for everything. Ollama supports only `max_completion_tokens`

vLLM started with `max_completion_tokens` but at some point switched to `max_tokens` and throws a harmless warning for the former. We kept both in GuideLLM for compatibility, but some model servers *cough cough Ollama* don't like that. As of now if you have an issue you can set `--backend_args="{'remove_from_body':['max_completion_tokens']}"` to remove it from the request. In the future we may support custom request templates and default to only having `max_tokens` (#194).
Repository: vllm-project/guidellm
Issue: Incorrect info in the "Running the UI" section of the README
Root cause: Documentation Deficiency

The README currently states:

> After running a benchmark with GuideLLM, a report.html file will be generated (by default at guidellm_report/report.html)

Running `guidellm benchmark` commands does not appear to create any html reports, only the default json file, or the csv or yaml file if specified on the command line.

It also points to a [hosted UI](https://neuralmagic.github.io/guidellm/ui/dev/) that gives a 404.

I believe the bits need for this just have not landed yet. See #191

@stmcginnis Sorry about that, thanks for pointing this out! Will change asap. This has not landed yet as @sjmonson said and the PR which implements it, #191, will be implementing it slightly differently. The html report will only be generated if the output-path arg is an html file.

But the hosted dev UI will live in the location you pointed out once the PR is merged in. Although that link will be updated to the prod UI.
Repository: vllm-project/guidellm
Issue: Bug: Negative TTFT values when benchmarking Ollama at high concurrency
Root cause: Validation Gap

When running a guidellm benchmark with --rate-type concurrent against an Ollama server at high concurrency rates, the resulting report can contain negative values for time_to_first_token_ms and other time-based metrics.

This seems to occur when the Ollama server is under heavy load and may not be returning a valid streaming response for some requests, causing an issue in how guidellm calculates the timing for those requests.

### To Reproduce
Set up an Ollama server and configure it for high parallelism (OLLAMA_NUM_PARALLEL=32).
Load a model such as llama3.1:8b-instruct-fp16.
Run a guidellm benchmark with a high concurrency rate (--rate 128).
Inspect the output JSON file.

### Faulty Output JSON Example: 

The following is an extract from a benchmarks.json file generated during a high-concurrency test, showing the negative values:

```
"time_to_first_token_ms": {
    "successful": {
        "mean": -1715507074412.1443,
        "median": -1750878354510.5388,
        "mode": -1750878498066.5781,
        "variance": 6.067968657158425e+22,
        "std_dev": 246332471614.24792,
        "min": -1750878498066.5781,
        "max": 89488.18135261536,
        "count": 99,
        "total_sum": -169835200366802.25
    }
}

```

A temporary patch was suggested by @sjmonson that adds error handling to the _iterative_completions_request function in src/guidellm/backend/openai.py.

This patch checks if first_iter_time or last_iter_time were ever set. If not (which happens when the server is overloaded and sends no data back for a request), it raises a ValueError instead of proceeding with faulty time calculations.

I can confirm that after applying this patch locally, the issue was resolved. Error requests were correctly marked as errors, and the benchmark only produced positive time values.

Here is the diff for the temporary fix:

```
diff --git a/src/guidellm/backend/openai.py b/src/guidellm/backend/openai.py
index 4eb6ae0..0748213 100644
--- a/src/guidellm/backend/openai.py
+++ b/src/guidellm/backend/openai.py
@@ -630,6 +630,9 @@ class OpenAIHTTPBackend(Backend):
                       response_prompt_count = usage["prompt"]
                       response_output_count = usage["output"]
 
+        if first_iter_time is None or last_iter_time is None:
+            raise ValueError("No iterations received for request: {}", request_id)
+
         logger.info(
             "{} request: {} with headers: {} and params: {} and payload: {} completed"
             "with: {}",
```



@sjmonson thanks for jumping in and proposing a fix, could we get a PR up and merged w/ this fix or do we need to do more investigation? 

Adding in to check this for the 0.4 release/refactor

@Harshith-umesh do you have any availability to validate if this is still happening now that 0.4.0?
Repository: vllm-project/guidellm
Issue: Input should be a valid dictionary or instance of StatusBreakdown
Root cause: External Dependency Breakage

Hi,
I installed guidellm from source (commit hash 91083695fbbc2fa95d81d95be3b6f5de415d9557).

I used vLLM to serve a model (llama 3.1 8B) and ran this guidellm command:
```
export MODEL_NAME=/hf_models/meta-llama/Meta-Llama-3.1-8B-Instruct/

guidellm benchmark \
  --target "http://localhost:8008" \
  --model $MODEL_NAME \
  --rate-type concurrent \
  --rate 4.0 \
  --max-seconds 15 \
  --data "prompt_tokens=256,output_tokens=16"
```

The benchmark seems to run for 15 seconds, but then fails:
```
pydantic_core._pydantic_core.ValidationError: 1 validation error for BenchmarkRunStats
requests_made
  Input should be a valid dictionary or instance of StatusBreakdown[int, int, int, int] [type=model_type, 
input_value=StatusBreakdown(successfu...incomplete=4, total=144), input_type=StatusBreakdown]
    For further information visit https://errors.pydantic.dev/2.9/v/model_type
```

The python/pydantic version I use:
```
python --version
Python 3.10.12

pip show pydantic
Name: pydantic
Version: 2.9.2
```

I tested this with the following Python and pydantic versions:
```
[root@ubi-playground guidellm]# python3 --version
Python 3.9.21
[root@ubi-playground guidellm]# pip show pydantic
Name: pydantic
Version: 2.11.7
```

And I did not get that validation error on both main and on the same commit, with the only major difference being that I used a different model. Your pydantic install is 2 minor versions behind. I wonder if that has anything to do with it. Can you retest this with a more modern version of pydantic?
Additional context: Mine also runs for about 15 seconds. The error appears to be related to the output processing.

I created a new python venv (python 3.10.12) with **pydantic-2.11.7** and install guidellm (commit hash https://github.com/vllm-project/guidellm/commit/91083695fbbc2fa95d81d95be3b6f5de415d9557).
The guidellm command **worked**.

I then downgraded pydantic:
```bash
pip install pydantic==2.9.2
...
      Successfully uninstalled pydantic-2.11.7
Successfully installed pydantic-2.9.2 pydantic-core-2.23.4
```

And got the error again:
```
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for BenchmarkRunStats
```

I then upgraded pydantic back to a new version:
```bash
pip install pydantic --upgrade
...
      Successfully uninstalled pydantic-2.9.2
Successfully installed pydantic-2.11.7 pydantic-core-2.33.2
```
And the command worked again.

So your observation seems to be correct, the issue seems to be related to **pydantic-2.9.2**.
Repository: vllm-project/guidellm
Issue: Different prompts generated for different concurrency levels despite same seed/config
Root cause: Algorithmic Error


## Summary
When running the concurrent benchmark across multiple concurrency levels (1, 2, 4, 8, 16, 32, 64), guidellm generates different prompts for each concurrency level, even when using the same seed/ synthetic data generation configuration. This makes it impossible to do fair "apples-to-apples" comparisons across different concurrency levels.

## Expected Behavior
When using the same seed and data configuration, all concurrency levels should generate the same set of prompts to ensure fair comparison across different concurrency levels.


## Evidence from Benchmark Results

Here's the analysis from a benchmark run with concurrency levels [1, 2, 4, 8, 16, 32, 64]:

### Configuration (identical for all benchmarks):
```json
{
  "request_loader": {
    "type_": "generative_request_loader",
    "data": "{\"prompt_tokens\":1000,\"prompt_tokens_min\":1,\"prompt_tokens_max\":4096,\"output_tokens\":1000,\"output_tokens_min\":1,\"output_tokens_max\":4096}",
    "processor": "meta-llama/Llama-3.1-8B-Instruct",
    "data_args": null,
    "processor_args": null
  }
}
```

### Different First Prompts by Concurrency Level:

- **Concurrency 1**: "such a sacrifice to her advantage as years of gratitude cannot enough acknowledge. By this time she is actually with them! If such goodness does not m..."

- **Concurrency 2**: ". How unlucky that you should have a reasonable answer to give, and that I should be so reasonable as to admit it! But I wonder how long you _would_ h..."

- **Concurrency 4**: "rejection. He who, she had been persuaded, would avoid her as his greatest enemy, seemed, on this accidental meeting, most eager to preserve the acqua..."

- **Concurrency 8**: "aunt now began; and each of them pronounced him to be infinitely superior to anything they had expected. He is perfectly well behaved,..."

- **Concurrency 16**: "you speak ill of a human being in my life. I would wish not to be hasty in censuring anyone; but I always speak what I think. I know you do and it is ..."

- **Concurrency 32**: "herself the reason, she had very little to say in reply. Mr. Gardiner left them soon after breakfast. The fishing scheme had been renewed the day befo..."

- **Concurrency 64**: ". Bingley did not take the house so much for the convenience of the neighbourhood as for his own, and we must expect him to keep or quit it on the sam..."



### Environment:
- Platform: OpenShift
- Backend: Ollama (llama3.1:8b-instruct-fp16)
- Data source: Synthetic text generation using Pride and Prejudice


## Reproduction Steps
1. Run a benchmark sweep with multiple concurrency levels:
   ```bash
   guidellm --target "<your llm server > --data {"prompt_tokens":1000,"prompt_tokens_min":1,"prompt_tokens_max":4096,"output_tokens":1000,"output_tokens_min":1,"output_tokens_max":4096} --model llama3.1:8b-instruct-fp16 --rate-type concurrent --rate 1,2,4,8,16,32,64
   ```
2. Examine the resulting benchmark file
3. Compare prompts across different concurrency levels



---
Repository: vllm-project/guidellm
Issue: [Feature Request] Need for ability to save partial results
Root cause: Unimplemented Feature Gap

As it stands, when run with multiple values to the rate parameter, there is nothing written to disk until the end of all the rates. This causes a potential for loss of all results when the higher concurrencies might potentially run into some error. Some checkpointing to the disk would greatly improve the usability of the tool when running into potential errors. 

The following is another scenario that we regularly encounter
1. We run a list of concurrencies. 
2. We observe KV Cache depletion on the server side. 
3. The latter(higher) concurrencies are of no real use. The user cannot choose to end the run. The runs have to complete to get any usable results. 

One note on this is that the error rate and over saturation detection implementations should fix the latter scenario
Repository: vllm-project/guidellm
Issue: The command "guidellm benchmark --help" does not work with latest click version
Root cause: External Dependency Breakage

I installed guidellm (from source) on a new python venv (python 3.10).
When I try to run `guidellm benchmark --help` I get this error:
```
  File "/.../python3.10/site-packages/click/core.py", line 2212, in make_metavar
    metavar = self.type.get_metavar(param=self, ctx=ctx)
TypeError: Union.get_metavar() got an unexpected keyword argument 'ctx'
```

This error is related to the `click` package, after a quick search I found that this issue is related to click version 8.
So I downgraded click to version 7:
```bash
pip install click==7.1.2
...
      Successfully uninstalled click-8.2.1
Successfully installed click-7.1.2
```

And the `--help` worked with that version, but then a `guidellm benchmark` command fails:
```
  File "/.../guidellm/src/guidellm/utils/cli.py", line 23, in set_if_not_default
    if ctx.get_parameter_source(k) != click.core.ParameterSource.DEFAULT:  # type: ignore[attr-defined]
AttributeError: 'Context' object has no attribute 'get_parameter_source'
```

So simply downgrading `click` is not a solution.

@pliops-daniels this issue has been fixed on `main`. Please downgrade to click `8.1` not `7.1.2`. (ref: #200)
Repository: vllm-project/guidellm
Issue: [Feature Request] Consider having groups of queries with multiple system prompts
Root cause: Unimplemented Feature Gap

This is related to the ongoing work for multi turn chat, for this suggestiion we have multiple system prompts along with a set of questions that go with that system prompt. This has been implemented in the sglang bench_serving.py modifications of the vllm version of this script (https://github.com/sgl-project/sglang/blob/main/python/sglang/bench_serving.py). The configuration knobs are listed below.

    group.add_argument(
        "--gsp-num-groups",
        type=int,
        default=64,
        help="Number of system prompt groups for generated-shared-prefix dataset",
    )
    group.add_argument(
        "--gsp-prompts-per-group",
        type=int,
        default=16,
        help="Number of prompts per system prompt group for generated-shared-prefix dataset",
    )
    group.add_argument(
        "--gsp-system-prompt-len",
        type=int,
        default=2048,
        help="Target length in tokens for system prompts in generated-shared-prefix dataset",
    )
    group.add_argument(
        "--gsp-question-len",
        type=int,
        default=128,
        help="Target length in tokens for questions in generated-shared-prefix dataset",
    )
    group.add_argument(
        "--gsp-output-len",
        type=int,
        default=256,
        help="Target length in tokens for outputs in generated-shared-prefix dataset",

There's a new pathway going in with 0.4 to treat all inputs as datasets with custom/default templates to combine. Prefixes will be enabled through this either in the 0.4 or quickly in the 0.5
Repository: vllm-project/guidellm
Issue: Enable dynamic Prefix Handling
Root cause: Unimplemented Feature Gap

This is a follow-up (Part-2) to the feature of adding unique prefixes for SyntheticTextItemsGenerator. In Part-1, we decided to add a controllable, different prefix during data generation ([see issue #186](https://github.com/vllm-project/guidellm/issues/186)). The next step is to separate the prefix handling from data generation, so the prefix can be generated, appended, and modified as needed for each backend request. This will help control the prefix cache rate more effectively.
Repository: vllm-project/guidellm
Issue: Over-Saturation stopping
Root cause: Unimplemented Feature Gap

@markurtz This issue is dependent on #238.
When running a benchmark there are many reasons that could cause the server to Over-Saturate, meaning that the rate of guidellm generated requests exceeds the rate of server responses.
Usually this causes a major skew in the measured metrics, making them false and misleading. 

The proposed feature is to integrate an online Over-Saturation detection algorithm into guidellm.
The algorithm will be used in 3 ways:
1. Early stopping a benchmark which detects Over-Saturation
2. Early stopping a sweep which detects Over-Saturation
3. Output report indication of the detection of Over-Saturation

## Over-Saturation Detection Algorithm
I have evaluated an algorithm (see internal RedHat slack for links and documents) which achieves near perfect detection, both in terms of accuracy and time waste minimization.
The algorithm basically goes like this:
1. Wait at least 30 seconds (hyper-param, tunable)
2. Wait for median TTFT to get larger than 2.5 seconds (hyper-param, tunable)
3. Check the slope of concurrent requests over time
4. Check the slope of TTFT over time
5. If both are consistently positive (reasonable margin of error) - stop the benchmark.

## Is this issue addressed by an existing feature?
Currently, Over-Saturation is partially addressed by:
1. Throughput/sweep mode measure the max load which is achievable by a server. In practice, the current throughput mode detected RPS is very noisy and usually over-estimates it by large, therefore a sweep actually usually Over-Saturates the server in the last few constant benchmarks.
2. Max error stopping #238 will also help when an Over-Saturated server starts not responding and errors accumulate, but it takes a while for timeouts to start coming, and that time is wasted.

## Implementation Discussion
It seems very natural to me to put this logic somewhere between the `benchmarker.py`, `aggregator.py` and potentially a new class:
https://github.com/vllm-project/guidellm/blob/72374efdf7d4432173fafec3924dc94ac3b11449/src/guidellm/benchmark/benchmarker.py#L225
The `aggregator.py` or the new class could accumulate the necessary information for calculating the slopes and margin of errors (TTFT and concurrent requests over some window of time, e.g 1m), and if an over-saturation is detected, then the `benchmarker.py` will send a stop signal to the scheduler, set the termination reason to be "over-saturated-server", complete the benchmark gracefully and if it is a sweep - break the profile strategy loop.
@markurtz, what do you think?

Fixed by #438 when that lands
Repository: vllm-project/guidellm
Issue: Margin Of Error (MOE) stopping
Root cause: Unimplemented Feature Gap

Given that #238 will land and that #243 will turn into a PR and land as well, it would be incredible to have target metric MOE stopping.
Basically this feature is 2 new options:
* `--target-metric` - the metric to use as the target for MOE stopping, of the form ["all", "ttft", "ttft-p90", "itl-median"....] (default: "all")
* `--target-moe` - the allowed error in the metric measurement (in percentages) as reflected in the MOE of that metric.
In a benchmark run, when the MOE value of the target metric reaches the target MOE - the benchmark will end with the termination reason "target_moe_reached".

This feature will include a simple ETA value.
Essentially, calculate the amount of samples required to reach the MOE (assuming the average sample weight consists), then use the RPS to extrapolate time.

A feature like this would make guidellm much faster and reliable for most use-cases.
Repository: vllm-project/guidellm
Issue: Requests Latency in (ms) --> (s)?
Root cause: Algorithmic Error

Hi I'm using the latest version of guidellm 0.2.1, I noticed that in the Benchmarks Stats the  Req Latency is in (ms) not (sec) as shown below. 

<img width="1636" height="224" alt="Image" src="https://github.com/user-attachments/assets/eb90bbfb-a20b-44db-9134-6c18cc4bf974" />

The slide from the office hours says otherwise(image below), and it makes sense that it should be in seconds, not ms, but just wanted to double check. Thanks!
<img width="1124" height="166" alt="Image" src="https://github.com/user-attachments/assets/eb1b2967-de57-4f7f-96a7-3dec131d2d52" />

Hi. This has been fixed on main and will be in the next release.
Repository: vllm-project/guidellm
Issue: UI latency reported units as ms- should be sec
Root cause: Validation Gap

Throughout the UI latency units are reporting as ms when the values are in seconds.

Hi @DaltheCow , could you take a look at https://github.com/vllm-project/guidellm/pull/256 ?

@tjandy98, thanks for fixing this! PR is approved, looks like the DCO check is failing due to a missing sign off in the commit. I think that is a blocker to merging.

The UI build is failing but that is expected, so it's not an issue.

Fixed from #256 is merged in
Repository: vllm-project/guidellm
Issue: [Feature Request] List as input for max-seconds and max-requests
Root cause: Unimplemented Feature Gap

We need the ability to do a list of max-seconds or max-requests. The different rates that we set on the --rates parameter need different number of requests or different test durations to get usable results from the test. 

For concurrency of 600, if we require 600 seconds, we are now forced to run concurrency 1 for 600 seconds too. 
The same applies if we were to attempt to use max-requests too. Concurrency 1 test does not require 600 or 1200 requests to get valid data.

~This will be enabled in 0.4~

Pushed
Repository: vllm-project/guidellm
Issue: No Feedback When Generating Random Data
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
- When generating a long prompt like the below:
```bash
guidellm benchmark --target=$URL --model=$MODEL --rate-type=concurrent --rate=50 --max-requests=50 --output-path=~/guidellm_results.json --processor=$MODEL --data='{"prompt_tokens":8000, "output_tokens":5000}'
```

We get stuck generating the random data for about 1 minute:

```bash
(app-root) bash-5.1$ guidellm benchmark --target=$URL --model=$MODEL --rate-type=concurrent --rate=50 --max-requests=50 --output-path=~/guidellm_results.json --processor=$MODEL --data='{"prompt_tokens":8000, "output_tokens":5000}'
Creating backend...
Backend openai_http connected to http://10.16.1.195:8000 for model                    
meta-llama/Meta-Llama-3-8B-Instruct.                                                  
Creating request loader...
```

**Describe the solution you'd like**
There should be some feedback during this period about what is happening. I thought there was a bug in vllm or in guidellm during this period

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.


Related to #270 

Synthetic data now generates during the benchmark.