zhiminy/EvalEng huggingface dataset:
Repository: vllm-project/guidellm
Issue: CI Failure when no tests are collected
Root cause: Validation Gap

**Describe the bug**
E2E tests fail on `main` because we currently have no "smoke" tests for E2E.

**Expected behavior**
The GitHub action should ignore failures due to 0 tests being collected. #469 contained a check for this but it does not work.

**Errors**
```
============================= test session starts ==============================
platform linux -- Python 3.10.19, pytest-8.2.2, pluggy-1.6.0 -- /home/runner/work/guidellm/guidellm/.tox/test-e2e/bin/python
cachedir: .tox/test-e2e/.pytest_cache
rootdir: /home/runner/work/guidellm/guidellm
configfile: pyproject.toml
plugins: anyio-4.11.0, respx-0.22.0, timeout-2.4.0, Faker-37.11.0, cov-5.0.0, mock-3.14.1, rerunfailures-14.0, asyncio-1.1.1
asyncio: mode=strict, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collecting ... collected 4 items / 4 deselected / 0 selected

============================ 4 deselected in 0.32s =============================
test-e2e: exit 5 (2.30 seconds) /home/runner/work/guidellm/guidellm> python -m pytest tests/e2e -m smoke pid=3129
  test-e2e: FAIL code 5 (34.88=setup[32.59]+cmd[2.30] seconds)
  evaluation failed :( (34.92 seconds)
Error: Process completed with exit code 5.
```

**Additional context**
Non-working bit here: https://github.com/vllm-project/guidellm/blob/765c875a7378d916fe22accd6b66371062a43d54/.github/actions/run-tox/action.yml#L27-L28
Repository: vllm-project/guidellm
Issue: Switch from PDM to uv for advanced dependency management
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
When we initially adopted PDM, `uv` was not capable of many of the advanced features we use such as multi-platform locks, pulling dependencies from non-standard repositories, and native `pylock.toml` support. Most of these problems have been solved and now function better in `uv` than PDM.

**Describe the solution you'd like**
Switch to `uv` in CI and container builds

**Additional context**
See [this guide](https://docs.astral.sh/uv/guides/integration/pytorch/#installing-pytorch) on managing the CPU version of pytorch which is one of our main reasons for using PDM in the build process.
Repository: vllm-project/guidellm
Issue: Per-worker concurrency is capped at 100
Root cause: Architectural Constraint

**Describe the bug**
Setting `GUIDELLM__MAX_WORKER_PROCESSES=1` results in a maximum of exactly 100 concurrency measured from server-side. Client-side reports full concurrency meaning that the bottleneck is likely after we launch the request thread.

**Expected behavior**
Actual max concurrency per-worker should be much higher.

**Environment**
Include all relevant environment information:
1. OS [e.g. Ubuntu 20.04]: Fedora Linux 42 (Container Image) on OCP 4.19
2. Python version [e.g. 3.12.2]: 3.13.7
3. GuideLLM version: v0.4.0

**To Reproduce**
Exact steps to reproduce the behavior:

```sh
export GUIDELLM__MAX_WORKER_PROCESSES=1
guidellm benchmark \
                --target http://localhost:8000
                --rate-type concurrent \
                --rate "128" \
                --max-seconds 120 \
                --data "prompt_tokens=256,output_tokens=128"
```

Observe from server-side that there are never more than 100 waiting + running requests.

**Additional context**
Behavior is constant with more workers. E.g. `GUIDELLM__MAX_WORKER_PROCESSES=2` results in a maximum concurrency of 200. Our default is 10 workers; 1,000+ concurrency is pretty rare for single-node tests.
Repository: vllm-project/guidellm
Issue: Support on TGI (Text Generation Inference)
Root cause: Documentation Deficiency

In the future, is it possible for guidellm to support TGI (Text Generation Inference)?

Guidellm supports TGI as well, we are just missing docs. Here is how to start a TGI server (thanks to @philschmid):
https://github.com/neuralmagic/guidellm/pull/43/files



Awesome, thanks
Repository: vllm-project/guidellm
Issue: 'guidellm:latest' not available for image platform (linux/arm64)
Root cause: External Dependency Breakage

**Is your feature request related to a problem? Please describe.**
I'd like to run the guidellm container image on aarch64 system
sut# lscpu | grep Model  <-- Model name: Ampere-1a
sut# podman run --rm  ghcr.io/vllm-project/guidellm:latest
WARNING: image platform (linux/amd64) does not match the expected platform **(linux/arm64)**
{"msg":"exec container process `/opt/app-root/bin/guidellm`: Exec format error","level":"error","time":"2025-12-04T18:58:30.772456Z"}

**Describe the solution you'd like**
Availability of (linux/arm64) image

**Describe alternatives you've considered**
Using 'pip install' provides aarch64 wheels-pkgs

**Additional context**
None


We cannot offer this yet as one of our optional dependencies for audio benchmarking is still working on their Linux ARM builds (see https://github.com/meta-pytorch/torchcodec/issues/569). The container is designed to offer all features out of the box and I do not want to publish official ARM builds until this is resolved. 

As a temporary solution, once #509 lands you can build the container yourself with `GUIDELLM_BUILD_EXTRAS=recommended` which will drop the extras groups for multi-modal benchmarking. E.g.

```sh
podman build \
    --platform linux/arm64 \
    --build-arg GUIDELLM_BUILD_TYPE=release \
    --build-arg GUIDELLM_BUILD_EXTRAS=recommended \
    -t guidellm:latest \
    -f Containerfile \
    .
```
Repository: vllm-project/guidellm
Issue: GuideLLM v0.4.0 'sweep' can shutdown vLLM-cpu Mode server
Root cause: External Dependency Breakage

**Describe the bug**
On a RHEL 9.4 Intel Sapphire Rapids server with container image 'vllm-cpu-release-repo:v0.12.0' -or- 'vllm-cpu-release-repo:v0.11.2' running, starting the following GuideLLM Workload will cause vllm to shutdown during step 1 (sync):

pi-28# guidellm benchmark --target http://localhost:8000 \
    --processor "$PWD/Models/Llama-3.2-1B-Instruct" \
    --rate-type sweep \
    --data "prompt_tokens=32,output_tokens=16"
**GuideLLM hangs** on first test (sync) & causes vLLM server to shutdown
(APIServer pid=1) INFO:     10.88.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=1) INFO:     Shutting down

**Expected behavior**
I don't believe GuideLLM workload should be able to shutdown a vllm server.
It would be acceptable, if the workload cannot be completed, that Guidellm timeout

**Environment**
Include all relevant environment information:
1. OS [e.g. Ubuntu 20.04]: Red Hat Enterprise Linux 9.4 (Plow)
2. kernel:  5.14.0-427.13.1.el9_4.x86_64
3. Python version [e.g. 3.12.2]: Python 3.11.7
4. guidellm version: 0.4.0
5. podman version 4.9.4-rhel

**To Reproduce**
Exact steps to reproduce the behavior:
console1# podman run --name vllm-cpu --rm --privileged=true    --shm-size=4g -p 8000:8000 -e VLLM_CPU_KVCACHE_SPACE=40    -v $PWD/Models:/model    public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.12.0   --model "/model/Llama-3.2-1B-Instruct" --dtype=bfloat16
<wait for Application Server to start>

console2# guidellm benchmark --target http://localhost:8000 \
    --processor "$PWD/Models/Llama-3.2-1B-Instruct" \
    --rate-type sweep \
    --data "prompt_tokens=32,output_tokens=16"

**Errors**
console1
GuideLLM hangs on first test (sync) & causes vLLM server to shutdown
(APIServer pid=1) INFO:     10.88.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=1) INFO:     Shutting down

console2 just hangs in GuideLLM output window, stuck on step 1 (sync)

**Additional context**
Add any other context about the problem here. Also include any relevant files.


This sounds like a broken vLLM build or some other non-GuideLLM issue. Under normal circumstances there is nothing a client can do to shut down the server.

I've used pre-built (and published) cpu-mode vLLM images, for both version v0.11.2 and v0.12.0, and have seen this behavior on several x86 systems.

I'm currently unable to test this scenario on ARM due to lack of docker images for vLLM on aarch64 (not published and unable to build locally). See:

- https://github.com/vllm-project/vllm/issues/29983
- https://github.com/vllm-project/vllm/issues/30076


Further testing does indicate a Problematic vLLM-cpu image is causing the failure.
CLOSING.
Repository: vllm-project/guidellm
Issue: OpenAI backend validation fails with 401 Unauthorized when API key authentication is required
Root cause: Validation Gap

**Describe the bug**
The OpenAI backend's validate() method fails when connecting to OpenAI-compatible API endpoints that require authentication on their health check endpoints. The validation request to the /health endpoint does not include the Authorization header, causing a 401 Unauthorized error even when a valid API key is provided.

**Expected behavior**
When using --backend-type openai_http with a valid API key, the backend validation should successfully authenticate against the remote server's health endpoint.

**Actual behavior**
httpx.HTTPStatusError: Client error '401 Unauthorized' for url 'https://my-server/health'
RuntimeError: Backend validation request failed. Could not connect to the server or validate the backend

**Environment**
Include all relevant environment information:
`guidellm version: 0.4.0`
`Python 3.12.12`

**To Reproduce**
Exact steps to reproduce the behavior:

1. Set environment variables for the key and base url (GUIDELLM__OPENAI__API_KEY, BASE_URL)
2. Run the benchmark

```
guidellm benchmark run \
  --target "$BASE_URL" \ 
  --backend-type openai_http \
  --model "model" \
  --rate-type constant --rate 1 \
  --max-requests 10 \
  --data 'prompt_tokens=256,output_tokens=128'
```

**Additional Context**
Refer to #491 


I have currently exactly the same issue.

Sorry, support for `GUIDELLM__OPENAI__API_KEY` was accidentally wiped out in v0.4.0. Assuming you don't need multi-modal you can switch to v0.3.1 and it should work. Will be fixed after #491.
Repository: vllm-project/guidellm
Issue: Can you list some popular HuggingFace datasets that we can used to set guidellm --data
Root cause: Documentation Deficiency

**What is the URL, file, or UI containing proposed doc change**
https://github.com/vllm-project/guidellm/blob/main/README.md

**What is the current content or situation in question**
GuideLLM supports HuggingFace datasets, local files, and synthetic data. This example loads the CNN DailyMail dataset from HuggingFace and maps the article column to prompts while using the summary token count column to determine output lengths.

```bash
guidellm benchmark \
  --target http://localhost:8000 \
  --data "hf:cnn_dailymail" \
  --data-args '{"prompt_column":"article","output_tokens_count_column":"summary_tokens"}'
```

**What is the proposed change**
Can you list some popular dataset that we can use, for example, used to calibrate MOE expert load balance. Or some guide to judge which types dataset we can use and which types of dataset can not.
Repository: vllm-project/guidellm
Issue: ModuleNotFoundError: No module named 'transformers.tokenization_utils'
Root cause: External Dependency Breakage

when using transformers 5.0.0rc0, there are some errors:

  File "/usr/local/lib/python3.12/dist-packages/guidellm/mock_server/handlers/chat_completions.py", line 35, in <module>
    from guidellm.mock_server.utils import (
  File "/usr/local/lib/python3.12/dist-packages/guidellm/mock_server/utils.py", line 17, in <module>
    from transformers.tokenization_utils import AddedToken, PreTrainedTokenizer, TextInput
ModuleNotFoundError: No module named 'transformers.tokenization_utils'


  File "/usr/local/lib/python3.12/dist-packages/guidellm/mock_server/handlers/tokenizer.py", line 16, in <module>
    from transformers.tokenization_utils import PreTrainedTokenizer
ModuleNotFoundError: No module named 'transformers.tokenization_utils'


Hi, a contribution is welcome but the GuideLLM team will not attempt to fix this until transformers 5.0.0 is out of pre-release.
Repository: vllm-project/guidellm
Issue: Get progress metrics in a file during run, not after the result is completed
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
GuideLLM currently writes benchmark metrics only after the run completes. During long-running benchmarks, progress metrics are available only via the interactive TTY UI, which is not usable in automated or non-TTY environments

**Describe the solution you'd like**
Add support for emitting progress metrics to a file periodically during the run (e.g., incremental JSON/JSONL output flushed at --output-sampling intervals), so progress can be monitored programmatically without requiring a TTY.

**Additional context**
This would significantly improve observability and automation for long-running benchmarks and production test pipelines.


@sjmonson @markurtz can you please help us here, or maybe guide us for some patchwork

The live metrics output to the console contains only a few counters and averages. If that is all you want then it should not be too hard to generate some periodic log messages. We are looking to improve logging especially in non-interactive ttys Generating the full metrics report without affecting benchmark performance could be a challenge so I would focus on the former first. 

To answer your second question a little better. The main benchmarking loop is [here](https://github.com/vllm-project/guidellm/blob/7666c658460bc34abe3cc821d3ca072cfd39074a/src/guidellm/benchmark/benchmarker.py#L133). You can look at the accumulator and the methods that take it as an argument for logging status.
Repository: vllm-project/guidellm
Issue: [bug] Model without system prompt fail
Root cause: Validation Gap

Hello, 

When trying to benchmark [google/gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it) i ran into an issue. It seems like `guidellm` uses `[system](https://github.com/neuralmagic/guidellm/blob/f7947110ae1446ce765d6019cc4ab0a31d335923/src/guidellm/backend/openai.py#L109)` message to sent request, which leads to 
```bash
Template error: syntax error: System role not supported (in <string>:1)
```

I tried to change to `user` but this leads to `pydantic` validation errors. Trying to debug this but any pointers or help is appreciated. 

Hey @philschmid, I landed the chang for role to user, are you still seeing an error pop up and if so do you have more details on it?

Nope I worked from the fork i had and it worked, will confirm from `main` 

Tested on commit works as well.
Repository: vllm-project/guidellm
Issue: Integration Testing Enablement / Expansion
Root cause: Unimplemented Feature Gap

**Description**
Ensure that the targeted files/packages have reasonable test coverage. This includes running the following tests:

**Requirements**

`backend.openai`
**Smoke Test:**
Utilize OpenAI token and API with model gpt-4o-mini.
Ensure startup and test_connection pass.
Submit a maximum of 3 requests testing minimal length prompts of various text with output_token_count set.
If output_token_count is not set, iterate through a limited number of token responses.
Check that the combined prompt in the text generation result is sensible and completed, and the prompt was passed through.

**Sanity Test:**
Utilize a local vLLM server with a tiny model and a local Llama.cpp server with a tiny model.
Run through the above-listed tests for OpenAI requests.

**Regression Test:**
Perform more extensive requests by spinning up a vLLM server on a GPU worker with a quantized Llama 3.1 8b.
Run the above tests for longer prompts and ensure completion.

`request.emulated`
**Smoke Test:**
Validate various config variations as shown in unit tests while combining with a llama-3.1 tokenizer (both instantiated and from str for a model name).
Ensure they generate the proper token distributions as expected.

**Sanity Test:**
None.

**Regression Test:**
Test other text sources with the tokenizer, such as another language text, and ensure the tokenizer runs through correctly and generates the expected output.
Additionally, test other tokenizers such as Mistral and Qwen.

`request.file`
**Smoke Test:**
Validate line-based file formats of text, jsonl, and csv with sample data while combining with a llama-3.1 tokenizer (both instantiated and from str for a model name).
Ensure they generate the proper token distributions as expected.

**Sanity Test:**
None.

**Regression Test:**
Validate json and yaml file formats with sample data while expanding to other tokenizers such as Mistral and Qwen.

`request.transformers`
**Smoke Test:**
Validate a handful of popular, small datasets on HF datasets, both loaded and passed in, as well as specified through a string.
Combine these with a llama-3.1 tokenizer (both instantiated and from str for a model name) and ensure they generate the proper token distributions as expected.

**Sanity Test:**
None.

**Regression Test:**
Test other tokenizers such as Mistral and Qwen.

`utils.injector`
**Smoke Test:**
Validate the injector functionality across the various stages of the guidellm UI (dev, staging, and prod).
Ensure that the desired data can be properly injected into the files and results in loadable HTML with the data contained and accessible.

**Sanity Test:**
None.

**Regression Test:**
Test other tokenizers such as Mistral and Qwen.

**Expectations**
Validation should confirm that all tests have sufficient coverage and meet the expectations as detailed above.
Report any gaps or failures in the coverage and provide suggestions for improvement.
Repository: vllm-project/guidellm
Issue: End-to-End Testing Enablement / Expansion
Root cause: Unimplemented Feature Gap

**Requirements:**

Files, test pathways, and expectations – validate and report that the targeted files/packages have reasonable test coverage after:
main.py

**Smoke Test:**  
Validate a minimal synchronous benchmark utilizing OpenAI token and API with model gpt-4o-mini with a 10 request limit and emulated data with minimal prompt and output size will run end to end and output the expected results

**Sanity Test:**
Validate various minimal benchmarks {sweep, synchronous, throughput, constant, poisson} with emulated data will run end to end and output the expected results for a tiny model on both a vLLM local server and a Llama.cpp local server limiting by request length or time and keeping to a managagable test length

**Regression Test:**
Validate various longer benchmarks {sweep, synchronous, throughput, constant, poisson} will run with the different data types and various settings for models, tokenizers, and other params
Repository: vllm-project/guidellm
Issue: how to call the remote api
Root cause: Documentation Deficiency

The remote server directory:
![7a3f1c5610cc98b06ecae5ebdad460c8](https://github.com/user-attachments/assets/a7014652-1284-4e6d-b071-f54e9bb56a4a)

The request url is like: http://10.10.10.10:40105/v1/chat/completions

Then I user the command:`guidellm \
  --target "http://10.10.10.10:40035/v1" \
  --model "MiniCPM3-4B" \
  --data-type emulated \
  --data "prompt_tokens=512,generated_tokens=128" \
  --rate-type sweep --rate 2
  --max-requests 2`

I got the error: Max retries exceeded with url: /MiniCPM3-4B/resolve/main/config.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1e0daaad70>, 'Connection to huggingface.co timed out. (connect timeout=10)'






















































@jackqdldd this looks like an issue for trying to load the tokenizer/processor. The processor is needed to calculate the token counts for the prompts to ensure the correct number are sent. You can prevent that by passing in --processor for the code on main and point it at a local directory with the tokenizer files in it or at an HF stub that has the desired tokenizer files. Otherwise, you can provide an HF dataset or a dataset file to load real data rather than synthetic which will prevent the processor from being loaded with the code now on main.
Repository: vllm-project/guidellm
Issue: Get Accuracy Metrics reported 
Root cause: Unimplemented Feature Gap

Eval Harness enablement - to be scoped and broken up further. 

The goal with this ticket is to leverage the [LLM Eval Harness ](https://github.com/EleutherAI/lm-evaluation-harness)code to tie in eval benchmarking into the performance benchmarking that is done by GuideLLM. 

Eval Benchmarking on public/private datasets generally can take a long time, we will have the user plug in how long they want to run a benchmark for. Can’t use a specific amount of time. - short, medium, long being the amount of time/ each benchmark can be a task.

The challenges here are developing subsets that are representative of the massive original datasets in order to get accurate benchmarks. Due to this reality, the first task here to be done by the research team is to split up these larger benchmark datasets into smaller, benchmarkable subsets so we can run evals in a matter of minutes vs. hours. 

Mark to lay out what we need there to extend out the backend.

Closing out as a duplicate
Repository: vllm-project/guidellm
Issue: problems with sglang
Root cause: Unimplemented Feature Gap

when I run guidellm with the default parameters:
'''
guidellm   --target "http://localhost:30020/v1/"   --model "meta-llama/Llama-3.1-8B-In
struct"   --data-type emulated   --data "prompt_tokens=112,generated_tokens=58"
'''

I got warnings that data is not available:

 | percentiles | WARNING - No data points available to calculate percentiles.
mean | WARNING - No data points available to calculate mean.

and finally, and Error:
FATAL: exception not rethrown
Aborted (core dumped)

The report is missing the metrics "Time to first token", "Inter token latency", "output token throughput"

any suggestion? I know the warnings are coming from the src/guidelln/core/ folder directory, specifically from the module distribution.py. I do not know why error is happening.

Notice there is no problem with vLLM

@eliecerecology we haven't explicitly tested or added support for SGLang. There are a few fixes landing with #91, though, that may fix issues here. 

Could you supply an example command for how you're launching SGLang?

Stale
Repository: vllm-project/guidellm
Issue: How to pass additional sampling parameters to the vllm server ?
Root cause: Documentation Deficiency

I would like to pass additional sampling parameters when invoking guidellm. Is it possible to do so ? 

#78 was recently merged which adds support for passing arguments to the backend when guidellm is used as a library.

@pinak-p let us know if #78 fixes the issues you were seeing. Otherwise can you provide an example for your scenario?

Thank you for adding that. I'll take a look and get back.

@markurtz @sjmonson  - How does one pass the argument from the cli ? I'm looking at passing my own parameters like `topk` or `ignore_eos` with the request.

> [@markurtz](https://github.com/markurtz) [@sjmonson](https://github.com/sjmonson) - How does one pass the argument from the cli ? I'm looking at passing my own parameters like `topk` or `ignore_eos` with the request.

Ah, we'll need to expand out a bit more for that use case. Currently the --backend-args are only passed into the constructor for the backend, not to each request.

Got it. I think it will be a useful feature to have for benchmarking specific scenarios. Also, using `ignore_eos` ensures the desired number of tokens are generated always.

@pinak-p apologies as this got lost in the backlog, but this has been actively supported for some time now. Use the request formatter kwargs pathway to control the request payloads