zhiminy/EvalEng huggingface dataset:
Repository: vllm-project/guidellm
Issue: GUIDELLM__MAX_CONCURRENCY is off by 1
Root cause: Algorithmic Error

The number of concurrent requests in throughput mode is always one less then `Settings.max_concurrency`.

**Example**

```sh
export GUIDELLM__MAX_CONCURRENCY="2"

guidellm --target http://localhost:8000/v1 \
         --model meta-llama/Llama-3.2-3B \
         --data-type emulated \
         --data prompt_tokens=512,generated_tokens=2048 \
         --rate-type throughput \
         --max-seconds 300
```

*Observe from server side that number of requests in queue is 1*

@parfeniukink is this something that you can quickly fix and push out? Thanks!

hey @sjmonson 
Could you please checkout the branch `70-guidellm__max_concurrency-is-off-by-1` for testing?
In the old version, we had this `max_concurrency` config, which does not affect concurrency. It appears only once in the code, and this is for limiting tasks generation in a `while` loop.

I created another CLI parameter `--workers` in https://github.com/neuralmagic/guidellm/pull/72.
Could you please tell me whether it's something you expected to see? Otherwise, provide more details about the expected behavior.

My assumption would be that `GUIDELLM__MAX_CONCURRENCY="2"` limits the number of concurrent async tasks to 2. Currently in main that limit seems off by 1; this seems like a simple bug?

I'm not sure what that `--workers` flag in #72 is for, is it intended to be a direct replacement for `Settings.max_concurrency`? ~Atm it does not seem to do anything.~

Edit: Appoligies I understand what `--workers` is doing now, it duplicates the main scheduling loop. I don't think this is something we want to do unless its spawned separate of the main thread (user-space threads or mp). But I don't think #72 is really related to this issue.

> My assumption would be that `GUIDELLM__MAX_CONCURRENCY="2"` limits the number of concurrent async tasks to 2. Currently in main that limit seems off by 1; this seems like a simple bug?
> 
> I'm not sure what that `--workers` flag in [#72](https://github.com/neuralmagic/guidellm/pull/72) is for, is it intended to be a direct replacement for `Settings.max_concurrency`? ~Atm it does not seem to do anything.~
> 
> Edit: Appoligies I understand what `--workers` is doing now, it duplicates the main scheduling loop. I don't think this is something we want to do unless its spawned separate of the main thread (user-space threads or mp). But I don't think [#72](https://github.com/neuralmagic/guidellm/pull/72) is really related to this issue.

Okay, now I understand you. Thanks for details. I'll back to you soon.
Repository: vllm-project/guidellm
Issue: Add flag to put guildellm into "non-interactive" mode
Root cause: Unimplemented Feature Gap

Guidellm's use of terminal control characters in its console output can cause issues for non-interactive consoles such as when run by another tool. This can be solved by adding a flag to disable clearing the screen between updates.

Closing out as #96 added --disable-progress which will remove the progress display that was clearing the screen. Additionally, --disable-console-outputs is added as well enabling minimal output to the console. Feel free to reping if you feel like that won't work / we need to expand past that.
Repository: vllm-project/guidellm
Issue: Client-side prompt token count is inaccurate
Root cause: Algorithmic Error

The client-side tokenization in guidellm fails to account for the extra tokens added in the server's chat prompt template. There are two possible workarounds: 

1. Enable usage metrics in each request and let the server tell us how many prompt tokens there are.
2. Use the `/completions` endpoint rather than `/chat/completions` as the chat template is not applied on the `/completions` endpoint.

#91 contains fixes towards both. There will be a bit of follow up work past that to enable the user further configuration on calculations and selecting which end point to request to
Repository: vllm-project/guidellm
Issue: Clean up JSON output
Root cause: Unimplemented Feature Gap

There is a lot of minor improvements that can be made to the JSON output of guidellm. For example:

1. Deduplicate the prompt field for each request
2. Flatten some field structures (e.g. `"decode_times": { "data": [] }` -> `"decode_times": []` ).
3. Label percentiles (e.g. `"request_latency_percentiles": [ 1, 2,... ]` -> `"request_latency_percentiles": { "p01": 1, "p05": 5,. ... }`)
4. Give max and min with all percentiles
5. Drop concurrences timestamps (possibly replace with percentiles/min/max/mean)

Feel free to dive in, but wanted to callout that there will be some restructuring along these lines for the output standardization in case there is duplicate work. Will share something out a bit later

Let's also make sure that the output JSON is storing all of the metadata we may want to know:
Model_name
Quantized (None, INT4, INT8, FP8)
Hardware
Inference Scenario
vllm version
vllm-config file (need the file)

GuideLLM results:
Tokens per Second 
Time to First Token (TTFT) 
Inter-token Latency (ITL)
End-to-End Request Latency (e2e_latency)
Requests Per Second (RPS) Profiles/Sweeps
Cost to generate a million output tokens (Internal) - Future


#91 fixes the issues @sjmonson brought up within this issue. @rgreenberg1 can we move the extra pieces you included here into it's own issue?
Repository: vllm-project/guidellm
Issue: Record partially completed request metrics
Root cause: Unimplemented Feature Gap

At the end of a test, guidellm cancels all pending requests. This causes the reported throughput to be lower depending on the portion of the overall test that the canceled requests cover. Instead we should record some kind of partial completions for the tokens completed within the deadline.

#96 enables us to now grab the intermediate state for requests that errored out. Will include the reporting of those pieces within that scope.

Closing out now that #96 has landed
Repository: vllm-project/guidellm
Issue: Base enablement to enable a dataset configuration file to be passed in rather than standard args, allow alias's for the folder structure
Root cause: Unimplemented Feature Gap

**Description:**
We will need to add support for GuideLLM to take in a dataset configuration profile as an argument (with the details of i/o tokens, for example) that can be used for benchmarking with GuideLLM. These dataset profiles and push them upstream to a folder under GuideLLM to then be pulled in and used as reference datasets for GuideLLM. 


**User Story:**
As a ml developer, I want to be able to provide standard dataset profiles, targeted at different use cases (RAG, Summarization, Chat...) and use that profile as a benchmarking target to better understand what my inference performance will look like for my model/use case when I move to production. 

**Additional Documentation:**

https://docs.google.com/document/d/1Ql6RI3_LbhQxqFCIu_n2A2CewK6OVVFZFYgu4KEjjtw/edit?usp=sharing 

**Acceptance criteria:**
Extend a new argument format for GuideLLM to accept a dataset profile as an argument. 
Dataset profile should have the following extensions and formats to be valid as arguments: 
- Text - .txt
- Audio - .wav
- image - .png or .jpg

@sjmonson you're taking this task on, correct?

Aside from that, general thoughts from my side:
1. We should set it up such that the API pathways support passing in configs / aliases for those configs into the APIs. Ex: guidellm --scenario <ALIAS | CONFIG_PATH>.
2. The aliases that are passed in will resolve based on the name of the config that would be stored within the code base such as src/guidellm/benchmark/scenarios
3. Within each config, it contains all required information for fully configuring the scenario including pointing at an HF dataset, locally stored file, a file accessible over HTTP, or synthetic data configuration. For the different dataset pathways, it should support the args for constructing the dataset; ex for synthetic data generation it should contain the source text, prompt tokens with distribution settings. It should specify how to decide on the number of output tokens / output token distribution to use including loading from the dataset, specifying the statistics for the output token distribution, or None. Finally, we should likely include a random seed for reproducibility pathways.

For point 3, a lot of that will become easier with the planned data pipelines rework with expansion to multi modality. Additionally, we'll need to consider what this is going to look like for expansion to multi modal as well.
Repository: vllm-project/guidellm
Issue: Text-dataset performance profile creation - decide what the profiles are. 
Root cause: Unimplemented Feature Gap

**Description:** 
 We will need to create the standardize profiles what sizing and create all of the relevant files/folders and land them in easily understandable, organized folder structures w/in GuideLLM to offer users the ability to benchmark different scenarios on GuideLLM. These profiles should cover the very specific cases that a performance team may want to run to understand model performance under various dataset loads. This set of data should be comprehensive and offer great flexibility in benchmarking models under a wide range of dataset breakdowns. 

**User Story**


**Additional Docs**


**Acceptance Criteria**
Create standard dataset profiles per: https://docs.google.com/document/d/1Ql6RI3_LbhQxqFCIu_n2A2CewK6OVVFZFYgu4KEjjtw/edit?usp=sharing 
Land these profiles in a 'dataset-profiles' folder under the GuideLLM upstream 
folder formatting: 
- README - describes the profiles below and the use cases they cover 
- SI/MO 
- SI/LO
- MI/MO 
- LI/SO
- LI/MO
- LI/LO 
- MI/XO 
- XI/MO
Repository: vllm-project/guidellm
Issue: Scope out and propose Community-oriented text dataset profiles for model validation
Root cause: Unimplemented Feature Gap

**Description:** 
 We will need to create the community-oriented dataset profiles and land them in understandable, organized folder structures w/in GuideLLM to offer users the ability to benchmark common community scenarios on GuideLLM. These profiles should cover common use cases that a developer may want to load test locally or in a dev environment to test the inference performance of their model for their desired use case.  This set of data should be Split up by use case and offer 1-5 community-oriented datasets so users can simulate loads they may expect to see in their applications. These should be named easily understandable profiles.  

Solidify the Inference Scenario Profiles for inference model validation. 

**Goal:** 
Get all of the relevant datasets that fit under the use case scenarios, run tokenizer on them to get the underlying the statistics. We will target synthetic data to actually use. Once this is scoped, we can create a follow up task on the details of what we need to implement and pass over from the Acceptance Criteria. 

**User Story**
As a new developer getting started with GuideLLM and performance benchmarking 

**Additional Docs**


**Acceptance Criteria**
Create standard dataset profiles per: https://docs.google.com/document/d/1Ql6RI3_LbhQxqFCIu_n2A2CewK6OVVFZFYgu4KEjjtw/edit?usp=sharing 
Land these profiles in a 'dataset-profiles' folder under the GuideLLM upstream 
folder formatting: 
- README - describes the profiles below and the use cases they cover 
- Chat
- - short_form_chat
- - mid_form_chat
- - long_form_chat
- Code Generation
- - 
- Summarization
- RAG 


@markurtz updated this ticket to be the scoping for the dataset profiles.
Repository: vllm-project/guidellm
Issue: Community-oriented audio dataset profile creation
Root cause: Unimplemented Feature Gap

**Description**
We will want to add support for audio datasets into GuideLLM to enable the benchmarking of audio multi-modal models like whisper. Since Audio datasets are fairly limited, we should aim to structure the data by use cases in a way where developers can easily understand the context of the data and what is being benchmarked by the model. 

**User Story**
As a developer, I want to benchmark a whisper model so that I can understand performance before I move to production with different audio dataset profiles to make sure my use case (call-center summarization, translation, etc. ) can be met on my target hardware. 

**Acceptance Criteria**
- Enable support for the leading hugging face supported audio datasets: https://huggingface.co/blog/audio-datasets#a-tour-of-audio-datasets-on-the-hub 
- Create profiles for the datasets for different use cases in structured folders for the use cases 
- - Miltilingual Language Translation
- - English Speech Recognition
- - Speech Translation
- - Audio Classification
- - TBD
Repository: vllm-project/guidellm
Issue: [Scoping] Enable Hardware Accelerator Monitoring metrics to be captured in benchmarks
Root cause: Unimplemented Feature Gap

**Description**
The purpose of this task is to scope which metrics we will want to be capturing from the Hardware Accelerators where the benchmarks are run and gather the technical requirements/libraries we will need to use to gather these metrics and get them displayed to the user. 

**Use Story**
As a developer, I Want to know the GPU metrics when I run benchmarks to understand if my server is over/underutilized for my projected workload. 

**Acceptance Criteria**
Create a follow up ticket with the list of required metrics we want to capture and the technical overview of what libraries we will use to capture and stream the metrics.
Repository: vllm-project/guidellm
Issue: Enable vLLM scheduler specific Metrics
Root cause: Unimplemented Feature Gap

**Description:** The purpose of this ticket is to surface vLLM specific scheduler metrics to provide users with a full array of data on the vLLM service running. 

**User Story:**
As a ml developer, I want to be able to see the vLLM specific scheduler metrics so that I can ensure that my benchmarks are taking full advantage of what vLLM has to offer and that my service is running optimally before I move to production. 

**Acceptance Criteria:** 

- Stream the metrics from Prometheus to the output format script: 
- Queue size
- KV cache usage
- Requests in Waiting, Swap, Running
- Create a new CLI output table dedicated for the vLLM specific metrics: 
- - needs further scoping
Repository: vllm-project/guidellm
Issue: Throughput run in sweep mode runs for too short of time
Root cause: Algorithmic Error

When running a sweep with `--max-seconds` set. The throughput mode runs for must less than `--max-seconds`. For example with `--max-seconds 300`, throughput ran for 65 seconds:

```md
│ [14:19:55]   100% synchronous           (0.02 req/sec avg)                                                                                     │
│ [14:24:55]   100% throughput            (0.53 req/sec avg)                                                                                     │
│ [14:26:00] ⠼  15% constant@0.08 req/s   (0.05 req/sec avg)
...
```

This seems to be caused by a request limit; setting `--max-requests 1000000` makes the issue disappear:

```md
╭─ Benchmarks ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ [14:41:58]   100% synchronous           (0.02 req/sec avg)                                                                                     │
│ [14:46:58]   100% throughput            (2.43 req/sec avg)                                                                                     │
│ [14:52:05] ⠋   0% constant@0.29 req/s   (0.09 req/sec avg)
```

It seems that the number of requests is limited by the number of successful requests in the previous synchronous mode run. 

https://github.com/neuralmagic/guidellm/blob/6379f42a50f34c10e1676df835700315e57a0a69/src/guidellm/executor/profile_generator.py#L327-L330
Repository: vllm-project/guidellm
Issue: Integrate lm-eval into GuideLLM
Root cause: Unimplemented Feature Gap

**Description**
We've talked about adding in [lm-eval ](https://github.com/EleutherAI/lm-evaluation-harness)to become a native pathway in GuideLLM to enable developers to run both benchmarks and accuracy evaluations in just one tool. This ticket will outline the necessary work to integrate lm-eval into GuideLLM. 

**Acceptance Criteria**
- Package up the lm-eval-harness library into GuideLLM
- Create new command: guidellm-eval that can then take the same input parameters as lm-eval:
- - model hf \
- - model_args pretrained=EleutherAI/gpt-j-6B \
- - tasks hellaswag \
- - device cuda:0 \
- - batch_size 8

@jennyyangyi-magic to scope this out a bit more and update
Repository: vllm-project/guidellm
Issue: Calculate "peak token throughput" number that is closer to vLLM generation throughput
Root cause: Unimplemented Feature Gap

When doing a GuideLLM sweep, the token throughput numbers for each RPS test are lower than a user would expect given the vLLM generation throughput metric. 

I think there are two main reasons for this. 
1. Server is under-utilized at the beginning of the run while the number of requests in flight is ramping up
2. GuideLLM undercounts the tokens generated towards the end of the test duration, due to cancelled requests not being counted (see https://github.com/neuralmagic/guidellm/issues/77) 

I propose that we add an additional performance metric, we could call it something along the lines of "peak token generation throughput", hopefully in fewer words.

I have a couple of ideas on how this could be calculated:
1. Calculate token throughput in a sliding window of 30s or 1 minute and use the max, using decode times across all ongoing requests in that window.
2. Calculate token throughput in the duration from the time that the first completed request finished, and the time last completed request started. This doesn't perfectly eliminate the "underutilization" at the beginning and end of the test, but gets pretty close.

WDYT?

#96 now enables us to grab the metrics for any responses that errored (though it makes an assumption on iterations == tokens, since we don't have usage stats from the server in this case). 

Since I need to rework the report generation, I'll include these fixes within that

Closing out now that #96 is on main
Repository: vllm-project/guidellm
Issue: Enable pass-through tags as an extra field to allow users to get custom metadata output
Root cause: Unimplemented Feature Gap

**Description**
The purpose of this feature is to enable a user to be able to pass in labels/Tags and go straight to the raw output so a user could pass this in and get in the output - extra field to allow users to get metadata output. This way a user can designate custom tag to be output in the CLI to pass along meta data in a pipeline to an output file or other application. 

**Acceptance Criteria**
- Enable a --metadata tag that takes a string so that the user can fill it in to pass it through to the output in the CLI. 

This was landed with #96 as --output-extras which supports any json input
Repository: vllm-project/guidellm
Issue: Have GuideLLM kick off a vLLM server automatically to avoid having the user install vLLM and assign the target themselves
Root cause: Unimplemented Feature Gap

**Description:**
The proposal here is to change the architecture of how GuideLLM runs so that when a user runs GuideLLM it automatically kicks off a vLLM server and supplies GuideLLM with the necessary data to run the benchmark. This also covers adding any necessary pass-through parameters to vLLM to GuideLLM so that the user can just run GuideLLM to do a full end-to-end benchmark on a model. This is a UX enhancement. 

**Acceptance Criteria:**
- Enable GuideLLM to kick off a vLLM server when GuideLLM is run
- Enable GuideLLM to accept the necessary pass-through arguments that need to go to vLLM:
-- model (required)
-- port (optional)
-- TBD

I'm not sure that we should add orchestration of the vLLM server to the scope of GuideLLM, for a few reasons:

- Want the tool to stay agnostic of the runtime engine, we may want to add different backends to support other runtime engines.
- Don't want the tool to be opinionated about the platform that the runtime engine is deployed on. If we wanted to support model deployment we would likely want to support k8s, KServe, Podman, bare-metal, etc...
- Want to avoid scope-creep and keep GuideLLM focused on it's current focus which is doing load tests against a pre-deployed model endpoint.

Alternatives:
- A separate project focused on simplifying model deployment that supports all platforms we care about (k8s Deployment, kserve, Podman, local [`vllm serve`]). We could even try to keep a set of "known-good" model configurations in that repo
- Refer users to existing model deployment guides / mechanisms.
Repository: vllm-project/guidellm
Issue: INFO log level is too verbose
GitHub labels: enhancement

good first issue

priority-low

internal

The INFO log level is way to verbose about the inner config of objects. We should push many of the messages to higher log levels and/or rewrite some messages to be less verbose.

Here is a proposal for what we could focus on in each log level:

* `ERROR`: Anything that leads to complete failure of the test
* `WARNING`: Things might be going wrong, or not happening in the expected way
    * Requests erroring
    * Concurrency saturated
* `INFO`: Things that should be happening that help explain the state
    * Info about run configuration, dataset...
    * What RPS settings are calculated in the sweep
    * Infrequent status updates
    * Info about output calculations / files that its dumped to
* `DEBUG`: Stuff like prompts and results
* `TRACE`: token-level events from the requests
Repository: vllm-project/guidellm
Issue: Land GuideLLM UI in Upstream Repository
Root cause: Unimplemented Feature Gap

**Description**
This ticket covers landing the GuideLLM local UI in the upstream repo and wiring it up properly so when a user runs a GuideLLM benchmark, the UI is able to viewed as a workload report to analyze the data. 

**Acceptance Criteria**
- Necessary code lands in this repo so that the UI can properly build. 
- Front-end UI maps to the back-end data generated from GuideLLM CLI.