Repository: vllm-project/guidellm
Issue: Up to 9% decrease in output tokens per second between v0.3 and v0.4
Root cause: Algorithmic Error

## **Describe the bug**

The measured output token throughput in GuideLLM v0.4.0 and later is significantly lower despite all other metrics being near identical.

## **Expected behavior**

Output token throughput should not differ from v0.3.x when all else is equal.

## **Environment**
Include all relevant environment information:
1. OS [e.g. Ubuntu 20.04]: Fedora 43 container on OCP 4.19
2. Python version [e.g. 3.12.2]: 3.13.9
3. GuideLLM version: 0.4.0 and 0.5.0

## **To Reproduce**

Run the following GuideLLM test on v0.3.0 and v0.4.0, then compare metrics in output JSON:

```sh
# Ensure we use the same endpoint across versions
export GUIDELLM_REQUEST_TYPE=text_completions
export GUIDELLM_TARGET=http://localhost:8080

guidellm benchmark run \
            --target="${GUIDELLM_TARGET}" \
            --model=RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \
            --processor=RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \
            --rate-type=concurrent \
            --data=prompt_tokens=1000,output_tokens=1000 \
            --max-seconds=600 \
            --rate=1,50,100,200,300,500,650
```

## **Additional context**

### Total Output Tokens per second over intended concurrency

<img width="1110" height="335" alt="Image" src="https://github.com/user-attachments/assets/da5361c2-a1dd-41c4-82f6-75fbd8d8fa24" />

### Percent Change from v0.3.0 baseline

<img width="355" height="1050" alt="Image" src="https://github.com/user-attachments/assets/d21e1087-00aa-4784-a484-29122e0079f3" />

Another place where the a regression is observable is measured request concurrency. P10 and lower suffer pretty heavily at 500 concurrency then improves at 650. This matches output token results.

#### Request Concurrency

  |   | mean | p001 | p01 | p05 | p10
-- | -- | -- | -- | -- | -- | --
version | concurrency |   |   |   |   |  
0.3.0 | 1 | 0.999876 | 1 | 1 | 1 | 1
0.4.0 | 1 | 0.999952 | 1 | 1 | 1 | 1
0.3.0 | 50 | 49.91858 | 16 | 50 | 50 | 50
0.4.0 | 50 | 48.928146 | 1 | 1 | 50 | 50
0.3.0 | 100 | 99.61601 | 16 | 100 | 100 | 100
0.4.0 | 100 | 97.151576 | 0 | 2 | 100 | 100
0.3.0 | 200 | 198.241756 | 14 | 118 | 200 | 200
0.4.0 | 200 | 196.219505 | 1 | 8 | 200 | 200
0.3.0 | 300 | 291.075135 | 16 | 118 | 239 | 239
0.4.0 | 300 | 293.583736 | 0 | 87 | 280 | 280
0.3.0 | 500 | 487.173261 | 10 | 95 | 485 | 500
0.4.0 | 500 | 440.305875 | 4 | 4 | 4 | 42
0.3.0 | 650 | 551.758179 | 15 | 82 | 82 | 82
0.4.0 | 650 | 563.084963 | 23 | 113 | 164 | 164


#### Request Concurrency % Change

  |   | mean | p001 | p01 | p05 | p10
-- | -- | -- | -- | -- | -- | --
version | concurrency |   |   |   |   |  
0.3.0 | 1 | nan% | nan% | nan% | nan% | nan%
0.4.0 | 1 | 0.01% | 0.00% | 0.00% | 0.00% | 0.00%
0.3.0 | 50 | nan% | nan% | nan% | nan% | nan%
0.4.0 | 50 | -1.98% | -93.75% | -98.00% | 0.00% | 0.00%
0.3.0 | 100 | nan% | nan% | nan% | nan% | nan%
0.4.0 | 100 | -2.47% | -100.00% | -98.00% | 0.00% | 0.00%
0.3.0 | 200 | nan% | nan% | nan% | nan% | nan%
0.4.0 | 200 | -1.02% | -92.86% | -93.22% | 0.00% | 0.00%
0.3.0 | 300 | nan% | nan% | nan% | nan% | nan%
0.4.0 | 300 | 0.86% | -100.00% | -26.27% | 17.15% | 17.15%
0.3.0 | 500 | nan% | nan% | nan% | nan% | nan%
0.4.0 | 500 | -9.62% | -60.00% | -95.79% | -99.18% | -91.60%
0.3.0 | 650 | nan% | nan% | nan% | nan% | nan%
0.4.0 | 650 | 2.05% | 53.33% | 37.80% | 100.00% | 100.00%



For Request Concurrency it seems that v0.3.0 is the inaccurate one. Recalculating concurrency from request start and end times bring the v0.3.0 percentiles much closer to v0.4.0:

#### Request Concurrency

  |   | count | mean | std | min | 0.1% | 1% | 5% | 10% | 25% | 50% | max
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
concurrency | version |   |   |   |   |   |   |   |   |   |   |  
1 | 0.3.0 | 120 | 0.5 | 0.502096 | 0 | 0 | 0 | 0 | 0 | 0 | 0.5 | 1
1 | 0.4.0 | 122 | 0.5 | 0.502062 | 0 | 0 | 0 | 0 | 0 | 0 | 0.5 | 1
5 | 0.3.0 | 3388 | 45.597403 | 4.849639 | 0 | 2 | 18 | 42 | 43 | 44 | 46 | 50
5 | 0.4.0 | 3396 | 46.004122 | 4.848149 | 0 | 2 | 17 | 43 | 44 | 45 | 47 | 50
100 | 0.3.0 | 4790 | 94.302088 | 11.010311 | 0 | 2.789 | 24.89 | 92 | 93 | 94 | 96 | 100
100 | 0.4.0 | 4798 | 94.668404 | 11.206696 | 0 | 2.797 | 24 | 92 | 93 | 95 | 97 | 100
200 | 0.3.0 | 5986 | 189.998496 | 28.175696 | 0 | 3 | 30 | 153.25 | 193 | 194 | 196 | 200
200 | 0.4.0 | 5992 | 190.29506 | 28.362782 | 0 | 3 | 30 | 151 | 193 | 195 | 197 | 200
300 | 0.3.0 | 5856 | 281.922643 | 51.564007 | 0 | 3 | 30.55 | 150.75 | 292 | 294 | 296 | 300
300 | 0.4.0 | 5946 | 282.198621 | 52.044279 | 0 | 3 | 30 | 150 | 292 | 295 | 297 | 300
500 | 0.3.0 | 6947 | 461.149417 | 101.111302 | 0 | 3.946 | 36 | 178 | 356 | 493 | 496 | 500
500 | 0.4.0 | 7003 | 461.991004 | 102.287375 | 0 | 4 | 35.02 | 175.1 | 350.2 | 495 | 497 | 500
650 | 0.3.0 | 6631 | 584.103755 | 150.971465 | 0 | 3.63 | 34.3 | 170.5 | 338 | 642 | 646 | 650
650 | 0.4.0 | 6823 | 585.782354 | 150.609707 | 0 | 3.822 | 34.22 | 171.1 | 342 | 644 | 647 | 650


#### Request Concurrency % Change

  |   | count | mean | std | min | 0.1% | 1% | 5% | 10% | 25% | 50% | max
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
concurrency | version |   |   |   |   |   |   |   |   |   |   |  
1 | 0.3.0 | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan%
0.4.0 | 1.67% | 0.00% | -0.01% | nan% | nan% | nan% | nan% | nan% | nan% | 0.00% | 0.00%
50 | 0.3.0 | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan%
0.4.0 | 0.24% | 0.89% | -0.03% | nan% | 0.00% | -5.56% | 2.38% | 2.33% | 2.27% | 2.17% | 0.00%
100 | 0.3.0 | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan%
0.4.0 | 0.17% | 0.39% | 1.78% | nan% | 0.29% | -3.58% | 0.00% | 0.00% | 1.06% | 1.04% | 0.00%
200 | 0.3.0 | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan%
0.4.0 | 0.10% | 0.16% | 0.66% | nan% | 0.00% | 0.00% | -1.47% | 0.00% | 0.52% | 0.51% | 0.00%
300 | 0.3.0 | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan%
0.4.0 | 1.54% | 0.10% | 0.93% | nan% | 0.00% | -1.80% | -0.50% | 0.00% | 0.34% | 0.34% | 0.00%
500 | 0.3.0 | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan%
0.4.0 | 0.81% | 0.18% | 1.16% | nan% | 1.37% | -2.72% | -1.63% | -1.63% | 0.41% | 0.20% | 0.00%
650 | 0.3.0 | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan% | nan%
0.4.0 | 2.90% | 0.29% | -0.24% | nan% | 5.29% | -0.23% | 0.35% | 1.18% | 0.31% | 0.15% | 0.00%