Repository: vllm-project/guidellm
Issue: GuideLLM Sweep Benchmark: Low KV Cache Utilization and Limited QPS on H100 GPUs
Root cause: Algorithmic Error


**Summary:**

GuideLLM is expected to be highly useful for benchmarking and evaluating our Inference Gateway project: [[Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension)](https://github.com/kubernetes-sigs/gateway-api-inference-extension). However, we need to operate at higher QPS with 6-10 backends to truly evaluate our performance effectively. The sweep benchmark on our 10 vLLM server setup (H100 GPUs in GKE) showed surprisingly low max KV cache use (around 25%) and low QPS (max 210 QPS vs. expected ~260). We need to figure out the best settings or fix any issues in GuideLLM's sweep mode to match our past results better.

---

### Detailed Benchmark Issue: Low KV Cache Utilization and QPS Limitations

**Setup:**

- 10 vLLM model servers on H100 80GB GPUs on GKE
- Kubernetes (GKE), c4-standard-192 machine with vLLM endpoints co-located in the same  cluster
- GuideLLM benchmark command running on the c4-standard-192 hardware:
  ```bash
  guidellm benchmark --target http://34.2.30.26:80 \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --processor meta-llama/Llama-3.1-8B-Instruct \
    --data '{"prompt_tokens":955,"output_tokens":162,"samples":1000,"prompt_tokens_stdev":76,"output_tokens_stdev":92,"prompt_tokens_min":4,"prompt_tokens_max":1024,"output_tokens_min":4,"output_tokens_max":1024}' \
    --rate-type sweep --rate 10 --max-seconds 300 \
    --warmup-percent 0.0 --cooldown-percent 0.0 \
    --output-path results/sweep_10_2025-04-24_20-09-17.json
  ```
- Set max concurrency (`GUIDELLM__MAX_CONCURRENCY`) to 16,900 (derived from previous benchmarking results of max 260 QPS at KV Cache saturation with 65 seconds max E2E latency)

**Issue Observed:**

- Only achieved ~25% KV cache utilization with GuideLLM sweep mode
- Maximum QPS observed was limited to 210 (expected ~260 based on previous benchmarks)
- Waiting queue size remained relatively low during tests confirming that throughput was not maximized

**Observations from Tests:**

The following figures illustrate:

1. **Request/s variation:** clearly shows how Request/s changes with the sweep, peaking around ~210 Request/s.

2. **KV Cache Utilization:** remained around 25%, indicating underutilization.

3. **Queue Waiting Size:** remained low throughout the test, showing no significant queue buildup.

# *kv cache utilization during the sweep*
<img width="935" alt="Image" src="https://github.com/user-attachments/assets/6c579ee9-95c0-4f8e-9061-2d7f6da6047c" />

# *qps obtained during the sweep*
<img width="1334" alt="Image" src="https://github.com/user-attachments/assets/c2c32f9a-6342-4c71-a8ad-18a72fbb119e" />

# *waiting queue size during the sweep*

<img width="945" alt="Image" src="https://github.com/user-attachments/assets/db1f9728-5766-4047-985e-1d74e71a98cc" />

**Additional Context:**

- Prior benchmarks conducted with `benchmarking_serving.py` from the vLLM repository achieved significantly higher KV cache utilization (100%) and QPS (~260). Current tests, despite being on identical hardware (c4-standard-192), are consistently falling short of these benchmarks.

**Action Requested:**

- Guidance on tuning or configuration adjustments to achieve expected performance by saturating KV Cache. 
- Clarification on sweep mode limitations or best practices
- Recommendations or documentation updates for achieving non-streaming mode benchmarks


Hi @kaushikmitr, thanks for bringing this up. Locally, I'll work on estimating maximum performance for the system (with a no-op server) to see what performance looks like in those cases. While I'm working on that, rather than running a sweep, can you try out a few things?
- Try running a constant rate benchmark at a few different levels up to the 260 rate you expect, and see if that is able to work correctly and not fall behind on rates. Specifically change out for the following with your command `--rate-type constant --rate 260`. 
- GUIDELLM__MAX_WORKER_PROCESSES defaults to 10; increasing this value should help avoid the GIL if any issues are arising from it. 
- Another setting to try would be GUIDELLM__MAX_ADD_REQUESTS_PER_LOOP, which defaults to 20. This is the number of requests that can be added at a time in the main loop, and increasing that will prioritize keeping the queues for the workers populated over active updates. 
- If you'd like to see more info displayed for the system, pass in the --display-scheduler-stats which will give you a look into the internals for delays and overheads -- you should see the overheads on the order of a few milliseconds. 
- You can also try turning off the output display and see if that changes anything -- it shouldn't based on the current architecture, but it would be good to double-check.

Finally, could you share the args and run_stats sections from your output file?

Sure here is the output: 
```
guidellm benchmark --target http://34.2.30.26:80 --model meta-llama/Llama-3.1-8B-Instruct --processor meta-llama/Llama-3.1-8B-Instruct --data '{"prompt_tokens":955,"output_tokens":162,"samples":1000,"prompt_tokens_stdev":76,"output_tokens_stdev":92,"prompt_tokens_min":4,"prompt_tokens_max":1024,"output_tokens_min":4,"output_tokens_max":1024}' --rate-type sweep --rate 10 --max-seconds 300 --warmup-percent 0.0 --cooldown-percent 0.0 --output-path results/sweep_10_2025-04-24_20-46-14.json
INFO:__main__:Scraping metrics for job vllm-podmonitoring in namespace default for duration 300
INFO:__main__:Sleeping for 300 seconds before next metrics scrape
Creating backend...
Backend openai_http connected to http://34.2.30.26:80 for model                 
meta-llama/Llama-3.1-8B-Instruct.                                               
Creating request loader...
Created loader with 1000 unique requests from                                   
{"prompt_tokens":955,"output_tokens":162,"samples":1000,"prompt_tokens_stdev":76
,"output_tokens_stdev":92,"prompt_tokens_min":4,"prompt_tokens_max":1024,"output
_tokens_min":4,"output_tokens_max":1024}.                                       
                                                                                
─ Benchmarks ─────────────────────────────────────────────────────────────────╮
│ [… synchr… (… Req:    0.8 req/s,    1.30s Lat,     1.0 Conc,     231 Comp, … │
│               Tok:  125.5 gen/s, 1600.9 tot/s,  41.7ms TTFT,    7.8ms ITL, … │
│ [… throug… (… Req:  204.3 req/s,   15.03s Lat,  3071.7 Conc,   62308 Comp, … │
│               Tok: 33468.5 gen/s, 420117.9 tot/s, 10907.1ms TTFT,   25.3ms … │
│ [… consta… (… Req:   26.2 req/s,    1.39s Lat,    36.4 Conc,    7848 Comp, … │
│               Tok: 4296.8 gen/s, 53816.5 tot/s,  41.9ms TTFT,    8.3ms ITL,… │
│ [… consta… (… Req:   51.6 req/s,    1.53s Lat,    79.0 Conc,   15471 Comp, … │
│               Tok: 8463.4 gen/s, 106053.5 tot/s,  51.5ms TTFT,    9.1ms ITL… │
│ [… consta… (… Req:   76.9 req/s,    1.70s Lat,   131.1 Conc,   23070 Comp, … │
│               Tok: 12628.4 gen/s, 158186.2 tot/s,  72.5ms TTFT,   10.0ms IT… │
│ [… consta… (… Req:  102.2 req/s,    1.97s Lat,   202.0 Conc,   30673 Comp, … │
│               Tok: 16784.8 gen/s, 210290.6 tot/s, 104.9ms TTFT,   11.5ms IT… │
│ [… consta… (… Req:  127.4 req/s,    2.22s Lat,   283.5 Conc,   38223 Comp, … │
│               Tok: 20913.0 gen/s, 262055.2 tot/s, 137.0ms TTFT,   12.8ms IT… │
│ [… consta… (… Req:  152.6 req/s,    2.60s Lat,   397.1 Conc,   45765 Comp, … │
│               Tok: 25025.1 gen/s, 313740.8 tot/s, 176.7ms TTFT,   14.9ms IT… │
│ [… consta… (… Req:  177.6 req/s,    3.14s Lat,   557.8 Conc,   53281 Comp, … │
│               Tok: 29134.3 gen/s, 365284.0 tot/s, 227.4ms TTFT,   17.9ms IT… │
│ [… consta… (… Req:  202.2 req/s,    4.67s Lat,   943.8 Conc,   60658 Comp, … │
│               Tok: 33124.1 gen/s, 415789.9 tot/s, 752.9ms TTFT,   24.0ms IT… │
╰──────────────────────────────────────────────────────────────────────────────╯
Generating... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ (10/10) [ 1:07:54 < 0:00:00 ]                    
                    
Benchmarks Metadata:
    Run id:9879eb29-bae8-4d64-81bd-851687a7787a
    Duration:4079.5 seconds
    Profile:type=sweep, strategies=['synchronous', 'throughput', 'constant',    
    'constant', 'constant', 'constant', 'constant', 'constant', 'constant',     
    'constant'], max_concurrency=None                                           
    Args:max_number=None, max_duration=300.0, warmup_number=None,               
    warmup_duration=0.0, cooldown_number=None, cooldown_duration=0.0            
    Worker:type_='generative_requests_worker' backend_type='openai_http'        
    backend_target='http://34.2.30.26:80'                                       
    backend_model='meta-llama/Llama-3.1-8B-Instruct'                            
    backend_info={'max_output_tokens': 16384, 'timeout': 300, 'http2': True,    
    'authorization': False, 'organization': None, 'project': None,              
    'text_completions_path': '/v1/completions', 'chat_completions_path':        
    '/v1/chat/completions'}                                                     
    Request Loader:type_='generative_request_loader'                            
    data='{"prompt_tokens":955,"output_tokens":162,"samples":1000,"prompt_tokens
    _stdev":76,"output_tokens_stdev":92,"prompt_tokens_min":4,"prompt_tokens_max
    ":1024,"output_tokens_min":4,"output_tokens_max":1024}' data_args=None      
    processor='meta-llama/Llama-3.1-8B-Instruct' processor_args=None            Extras:None 

Benchmarks Info:================================================================================
=============================================================================== 
Metadata                                        |||| Requests Made ||| Prompt   
Tok/Req   ||| Output Tok/Req   ||| Prompt Tok Total     ||| Output Tok Total    
||                                                                              
      Benchmark| Start Time| End Time| Duration (s)|  Comp|  Inc| Err|  Comp|   
Inc|   Err|  Comp|   Inc|   Err|     Comp|     Inc|  Err|     Comp|    Inc| Err 
---------------|-----------|---------|-------------|------|-----|----|------|---
---|------|------|------|------|---------|--------|-----|---------|-------|---- 
    synchronous|   20:46:40| 20:51:40|        300.0|   231|    1|   0| 956.8|   
979.0|   0.0| 162.7|  65.0|   0.0|   221015|     979|    0|    37586|     65|   
0                                                                               
     throughput|   20:51:40| 20:56:45|        305.0| 62308| 3155|   2| 946.8|   
945.7| 984.5| 163.8| 150.3| 127.5| 58991835| 2983729| 1969| 10207316| 474351|   
255                                                                             
 constant@26.21|   21:02:01| 21:07:01|        300.0|  7848|   39|   2| 946.8|   
931.7| 900.0| 164.2| 117.4| 190.0|  7430867|   36336| 1800|  1288875|   4579|   
380                                                                             
 constant@51.65|   21:07:23| 21:12:23|        300.0| 15471|   75|   1| 946.7|   
942.6| 991.0| 164.1| 106.9| 164.0| 14645795|   70697|  991|  2538948|   8014|   
164                                                                             
 constant@77.09|   21:13:10| 21:18:10|        300.0| 23070|  134|   1| 946.8|   
952.3| 945.0| 164.2| 112.6| 175.0| 21843623|  127614|  945|  3788240|  15090|   
175                                                                             
constant@102.54|   21:19:29| 21:24:29|        300.0| 30673|  189|   0| 946.8|   
942.0|   0.0| 164.2| 106.2|   0.0| 29040264|  178042|    0|  5035279|  20076|   
0                                                                               
constant@127.98|   21:26:14| 21:31:14|        300.0| 38223|  294|   2| 946.8|   
946.5| 906.0| 164.1| 114.2|  95.0| 36189889|  278258| 1812|  6273817|  33588|   
190                                                                             
constant@153.42|   21:33:28| 21:38:28|        300.0| 45765|  412|   1| 946.8|   
944.5| 930.0| 164.0| 109.5| 197.0| 43329188|  389129|  930|  7507355|  45114|   
197                                                                             
constant@178.86|   21:41:17| 21:46:17|        300.0| 53281|  554|   0| 946.8|   
944.8|   0.0| 164.0| 109.0|   0.0| 50447497|  523411|    0|  8740026|  60407|   
0                                                                               
constant@204.30|   21:49:34| 21:54:34|        300.0| 60658|  834|   1| 946.8|   
945.2| 906.0| 163.8| 112.0| 297.0| 57429430|  788311|  906|  9937090|  93429|   
297                                                                             
================================================================================
=============================================================================== 
                 
                 
Benchmarks Stats:
================================================================================
===========================================================================     
Metadata       | Request Stats         || Out Tok/sec| Tot Tok/sec| Req Latency 
(ms)  ||| TTFT (ms)              ||| ITL (ms)        ||| TPOT (ms)       ||     
      Benchmark| Per Second| Concurrency|        mean|        mean|  mean|      
median|   p99|    mean|  median|     p99| mean| median|  p99| mean| median|  p99
---------------|-----------|------------|------------|------------|------|------
-|------|--------|--------|--------|-----|-------|-----|-----|-------|-----     
    synchronous|       0.77|        1.00|       125.5|      1600.9|  1.30|      
1.26|  2.90|    41.7|    41.3|    48.5|  7.8|    7.8|  7.9|  7.7|    7.7|  7.8  
     throughput|     204.30|     3071.70|     33468.5|    420117.9| 15.03|      
15.15| 21.02| 10907.1| 11126.4| 13218.9| 25.3|   25.1| 36.2| 25.2|   25.0| 35.8 
 constant@26.21|      26.16|       36.35|      4296.8|     53816.5|  1.39|      
1.39|  3.09|    41.9|    41.6|    50.2|  8.3|    8.2|  8.7|  8.2|    8.2|  8.7  
 constant@51.65|      51.57|       79.03|      8463.4|    106053.5|  1.53|      
1.53|  3.41|    51.5|    47.7|    90.5|  9.1|    9.1| 10.0|  9.0|    9.0|  9.9  
 constant@77.09|      76.91|      131.05|     12628.4|    158186.2|  1.70|      
1.70|  3.78|    72.5|    64.4|   181.6| 10.0|   10.0| 11.4|  9.9|    9.9| 11.3  
constant@102.54|     102.25|      201.95|     16784.8|    210290.6|  1.97|      
1.97|  4.36|   104.9|    85.9|   304.8| 11.5|   11.4| 13.8| 11.4|   11.4| 13.6  
constant@127.98|     127.41|      283.45|     20913.0|    262055.2|  2.22|      
2.21|  4.90|   137.0|   107.4|   457.9| 12.8|   12.7| 16.0| 12.7|   12.7| 15.9  
constant@153.42|     152.55|      397.05|     25025.1|    313740.8|  2.60|      
2.59|  5.73|   176.7|   142.1|   541.6| 14.9|   14.8| 19.0| 14.8|   14.7| 18.8  
constant@178.86|     177.61|      557.82|     29134.3|    365284.0|  3.14|      
3.12|  6.91|   227.4|   184.1|   828.1| 17.9|   17.7| 23.5| 17.8|   17.7| 23.3  
constant@204.30|     202.20|      943.83|     33124.1|    415789.9|  4.67|      
4.55| 11.21|   752.9|   373.0|  5344.7| 24.0|   24.0| 33.5| 23.9|   23.9| 33.2  
================================================================================
===========================================================================     
                           
Saving benchmarks report...
Benchmarks report saved to results/sweep_10_2025-04-24_20-46-14.json
                      
Benchmarking complete.

```


 there were 12 errors either 503 or 429, not showing thse logs. I will try to run another benchmark with the settings as you suggested @markurtz 






@kaushikmitr any updates on if you were able to achieve a higher rate here? Also, if you could provide the output from the json file for those sections, that will help more since it logs out the system metrics for those. 

Also, over the weekend I ran some performance testing with a no op server and for the default settings GuideLLM was maxing out around 1000 requests per second. I'll see if I can better simulate the settings you have based on the outputs to see if the inflight requests are creating the issue -- I don't believe they should since you mentioned that the vLLM benchmarking script is going higher. My best guess is that the throughput benchmark is leading to inefficiencies both locally as well as on the distributed system so it's not actually measuring the maximum possible there and setting a constant rate will enable you to go higher.

Also, the no op server did uncover some issues where the progress display is lagging the actual benchmarks significantly as well as some initial requests failing, will get a fix up for those pieces and see if I can get an e2e test landed for performance with that no op server

Thanks, I am out till mid May, so have not had time to investigate this further, if you have an updated PR happy to test it once I am back. I tried constant rate but could not go above 220 or so, but don't have the logs handy. Can share more once I am back. We are particularly interested in the sweep as it allows us to automatically discover the inflection point near kv cache saturation. 

Note: need to double check if this is fixed by #251 after that lands

This should be fixed now with the 0.3