Repository: vllm-project/guidellm
Issue: Possible hang/deadlock in-between a rate change with constant rate-type
Root cause: Resource Mishandling

**Describe the bug**

I was running a vllm benchmark today and it stalled completely at one point (had to ^C to restart) - I think the hang was in guidellm (rationale below).  I re-ran the test and it completed, so likely a race condition / deadlock I guess.

**Environment**
Include all relevant environment information:
1. OS: Centos stream 9.6
2. Python version: 3.12.2
3. vLLM version: 0.11.0
4. GuideLLM version: 0.3.1

**To Reproduce**

  I was testing a single instance of vLLM on a 2 GPU setup, with this invocation:
```
vllm serve Qwen/Qwen3-0.6B --tensor-parallel-size 2 --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both","kv_connector_extra_config":{"num_cpu_blocks":64000}}' --gpu-memory-utilization=0.2 
```

guidellm command line is in the console output below.

**Errors**
No errors but the guidellm process completely stopped sending requests -the UI continued updating the little spinners, with no values changing until interrupted after a couple of minutes.  The vLLM instance continued accepting requests during this time, continued responding to metrics requests, but all metric values went to zero (indicating no new requests arriving) - hence I think guidellm is more likely to be the failure point here.

**Additional context**

Here's some verbose context I captured before restart, with py-spy poking at a few of the guidellm processes.  I'm pretty sure it was transitioning between the completion of rate=100 and starting rate=200 at the time (or perhaps 200 -> 500).

```
(vllm) [root@nathans-llm-d-2 prompt-500-output-100]# ps -ef |grep guidellm
root      261063   64438 12 04:15 pts/1    00:02:10 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280011  261063 81 04:28 pts/1    00:04:00 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280022  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280026  261063  2 04:28 pts/1    00:00:06 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280030  261063  2 04:28 pts/1    00:00:06 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280034  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280038  261063  2 04:28 pts/1    00:00:06 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280042  261063  2 04:28 pts/1    00:00:06 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280046  261063  2 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280050  261063  2 04:28 pts/1    00:00:06 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280054  261063  2 04:28 pts/1    00:00:06 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280058  261063  2 04:28 pts/1    00:00:06 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280062  261063  2 04:28 pts/1    00:00:06 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280066  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280070  261063  2 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280074  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280078  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280082  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280086  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280090  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280094  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280098  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280102  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280106  261063  2 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280110  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280114  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280118  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280122  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280126  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280130  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280134  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280138  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      280142  261063  1 04:28 pts/1    00:00:05 /source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000
root      285937   77590  0 04:33 pts/3    00:00:00 grep --color=auto guidellm
(vllm) [root@nathans-llm-d-2 prompt-500-output-100]# pip install py-spy
Collecting py-spy
  Downloading py_spy-0.4.1-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (510 bytes)
Downloading py_spy-0.4.1-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (2.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.8/2.8 MB 114.3 MB/s  0:00:00
Installing collected packages: py-spy
Successfully installed py-spy-0.4.1

(vllm) [root@nathans-llm-d-2 prompt-500-output-100]# py-spy top --pid 261063


Collecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000' (python v3.12.11)
Total Samples 800
GIL: 9.00%, Active: 9.00%, Threads: 5

  %Own   %Total  OwnTime  TotalTime  Function (filename)                                              
  1.00%   1.00%   0.030s    0.070s   render (rich/markup.py)
  1.00%   1.00%   0.020s    0.020s   normalize (rich/measure.py)
  0.00%   0.00%   0.020s    0.060s   wrap (rich/text.py)
  2.00%   2.00%   0.020s    0.020s   update_width (rich/console.py)
  0.00%   0.00%   0.020s    0.020s   cell_len (rich/cells.py)
  0.00%   9.00%   0.020s    0.380s   split_and_crop_lines (rich/segment.py)
  0.00%   0.00%   0.020s    0.020s   pick_bool (rich/_pick.py)
  1.00%   3.00%   0.020s    0.100s   rich_cast (rich/protocol.py)
  1.00%   1.00%   0.020s    0.040s   render (rich/text.py)
  0.00%   9.00%   0.020s    0.370s   render (rich/console.py)
  0.00%   0.00%   0.010s    0.010s   __add__ (rich/style.py)
  0.00%   0.00%   0.010s    0.010s   <lambda> (<string>)
  0.00%   0.00%   0.010s    0.020s   get_current_style (rich/text.py)
  0.00%   0.00%   0.010s    0.010s   strip_control_codes (rich/control.py)
  0.00%   0.00%   0.010s    0.010s   pop_style (rich/markup.py)
  0.00%   0.00%   0.010s    0.010s   extend (rich/containers.py)
  0.00%   1.00%   0.010s    0.020s   _get_cells (rich/table.py)
  0.00%   4.00%   0.010s    0.220s   _render (rich/table.py)
  0.00%   0.00%   0.010s    0.010s   _emoji_replace (rich/_emoji_replace.py)
  0.00%   0.00%   0.010s    0.010s   join (rich/text.py)
  1.00%   1.00%   0.010s    0.010s   add_row (rich/table.py)
  1.00%   1.00%   0.010s    0.010s   get_padding (rich/table.py)
  0.00%   0.00%   0.010s    0.010s   justify (rich/containers.py)
  1.00%   3.00%   0.010s    0.060s   _measure_column (rich/table.py)
  0.00%   0.00%   0.010s    0.020s   adjust_line_length (rich/segment.py)
  0.00%   0.00%   0.010s    0.020s   _parse (rich/markup.py)
  0.00%   9.00%   0.010s    0.370s   render_lines (rich/console.py)
  0.00%   1.00%   0.000s    0.070s   <genexpr> (rich/progress.py)
  0.00%   0.00%   0.000s    0.010s   __init__ (rich/text.py)
  0.00%   7.00%   0.000s    0.280s   __rich_console__ (rich/table.py)
  0.00%   0.00%   0.000s    0.010s   split (rich/text.py)

Press Control-C to quit, or ? for help.


(vllm) [root@nathans-llm-d-2 prompt-500-output-100]# py-spy top --pid 280142

Collecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000' (python v3.12.11)
Total Samples 2300
GIL: 0.00%, Active: 6.00%, Threads: 33

  %Own   %Total  OwnTime  TotalTime  Function (filename)                                              
  6.00%   6.00%   0.930s    0.930s   _recv (multiprocessing/connection.py)
  0.00%   6.00%   0.000s    0.930s   <module> (guidellm)
  0.00%   6.00%   0.000s    0.930s   benchmark_with_scenario (guidellm/benchmark/entrypoints.py)
  0.00%   6.00%   0.000s    0.930s   _Popen (multiprocessing/context.py)
  0.00%   6.00%   0.000s    0.930s   invoke (click/core.py)
  0.00%   6.00%   0.000s    0.930s   benchmark_generative_text (guidellm/benchmark/entrypoints.py)
  0.00%   6.00%   0.000s    0.930s   _callmethod (multiprocessing/managers.py)
  0.00%   6.00%   0.000s    0.930s   _bootstrap (multiprocessing/process.py)
  0.00%   6.00%   0.000s    0.930s   _spawn_process (concurrent/futures/process.py)
  0.00%   6.00%   0.000s    0.930s   run_until_complete (asyncio/base_events.py)
  0.00%   6.00%   0.000s    0.930s   __init__ (multiprocessing/popen_fork.py)
  0.00%   6.00%   0.000s    0.930s   process_loop_asynchronous (guidellm/scheduler/worker.py)
  0.00%   6.00%   0.000s    0.930s   _start_executor_manager_thread (concurrent/futures/process.py)
  0.00%   6.00%   0.000s    0.930s   run (guidellm/scheduler/scheduler.py)
  0.00%   6.00%   0.000s    0.930s   recv (multiprocessing/connection.py)
  0.00%   6.00%   0.000s    0.930s   run (asyncio/runners.py)
  0.00%   6.00%   0.000s    0.930s   _process_worker (concurrent/futures/process.py)
  0.00%   6.00%   0.000s    0.930s   run_in_executor (asyncio/base_events.py)
  0.00%   6.00%   0.000s    0.930s   __call__ (click/core.py)
  0.00%   6.00%   0.000s    0.930s   submit (concurrent/futures/process.py)
  0.00%   6.00%   0.000s    0.930s   _run_once (asyncio/base_events.py)
  0.00%   6.00%   0.000s    0.930s   _start_processes (guidellm/scheduler/scheduler.py)
  0.00%   6.00%   0.000s    0.930s   _launch (multiprocessing/popen_fork.py)
  0.00%   3.00%   0.000s    0.480s   get_nowait (<string>)
  0.00%   6.00%   0.000s    0.930s   _recv_bytes (multiprocessing/connection.py)
  0.00%   6.00%   0.000s    0.930s   run (guidellm/__main__.py)
  0.00%   6.00%   0.000s    0.930s   run (guidellm/benchmark/benchmarker.py)
  0.00%   6.00%   0.000s    0.930s   _process_runner (guidellm/scheduler/worker.py)
  0.00%   3.00%   0.000s    0.450s   is_set (multiprocessing/managers.py)
  0.00%   6.00%   0.000s    0.930s   main (click/core.py)
  0.00%   6.00%   0.000s    0.930s   run (multiprocessing/process.py)

Press Control-C to quit, or ? for help.


(vllm) [root@nathans-llm-d-2 prompt-500-output-100]# py-spy top --pid 280134

Collecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tCollecting samples from '/source/vllm/.venv/bin/python /source/vllm/.venv/bin/guidellm benchmark --output-sampling 0 --rate-type concurrent --target http://localhost:8000 --max-seconds 120 --data prompt_tokens=500,output_tokens=100 --rate 99,5,10,20,50,100,200,500,1000,2000' (python v3.12.11)
Total Samples 800
GIL: 0.00%, Active: 5.00%, Threads: 33

  %Own   %Total  OwnTime  TotalTime  Function (filename)                                              
  5.00%   5.00%   0.240s    0.240s   _recv (multiprocessing/connection.py)
  0.00%   5.00%   0.000s    0.240s   _bootstrap (multiprocessing/process.py)
  0.00%   5.00%   0.000s    0.240s   <module> (guidellm)
  0.00%   5.00%   0.000s    0.240s   start (multiprocessing/process.py)
  0.00%   5.00%   0.000s    0.240s   recv (multiprocessing/connection.py)
  0.00%   5.00%   0.000s    0.240s   _run_once (asyncio/base_events.py)
  0.00%   5.00%   0.000s    0.240s   submit (concurrent/futures/process.py)
  0.00%   1.00%   0.000s    0.060s   get_nowait (<string>)
  0.00%   5.00%   0.000s    0.240s   _recv_bytes (multiprocessing/connection.py)
  0.00%   5.00%   0.000s    0.240s   _run (asyncio/events.py)
  0.00%   4.00%   0.000s    0.180s   is_set (multiprocessing/managers.py)
  0.00%   5.00%   0.000s    0.240s   run (multiprocessing/process.py)
  0.00%   5.00%   0.000s    0.240s   _callmethod (multiprocessing/managers.py)
  0.00%   5.00%   0.000s    0.240s   __init__ (multiprocessing/popen_fork.py)
  0.00%   5.00%   0.000s    0.240s   invoke (click/core.py)
  0.00%   5.00%   0.000s    0.240s   run (guidellm/__main__.py)
  0.00%   5.00%   0.000s    0.240s   _Popen (multiprocessing/context.py)
  0.00%   5.00%   0.000s    0.240s   _launch_processes (concurrent/futures/process.py)
  0.00%   5.00%   0.000s    0.240s   process_loop_asynchronous (guidellm/scheduler/worker.py)
  0.00%   5.00%   0.000s    0.240s   run (guidellm/benchmark/benchmarker.py)
  0.00%   5.00%   0.000s    0.240s   run (guidellm/scheduler/scheduler.py)
  0.00%   5.00%   0.000s    0.240s   _launch (multiprocessing/popen_fork.py)
  0.00%   5.00%   0.000s    0.240s   run (asyncio/runners.py)
  0.00%   5.00%   0.000s    0.240s   run_until_complete (asyncio/base_events.py)
  0.00%   5.00%   0.000s    0.240s   _process_worker (concurrent/futures/process.py)
  0.00%   5.00%   0.000s    0.240s   _spawn_process (concurrent/futures/process.py)
  0.00%   5.00%   0.000s    0.240s   main (click/core.py)
  0.00%   5.00%   0.000s    0.240s   _process_runner (guidellm/scheduler/worker.py)
  0.00%   5.00%   0.000s    0.240s   __call__ (click/core.py)
  0.00%   5.00%   0.000s    0.240s   benchmark_generative_text (guidellm/benchmark/entrypoints.py)
  0.00%   5.00%   0.000s    0.240s   _start_processes (guidellm/scheduler/scheduler.py)

Press Control-C to quit, or ? for help.
```

I stopped after these few, obviously there were many other processes involved - I didn't look at all but the waiting-in-_recv pattern repeats in each child processes above, perhaps that helps.  Let me know if there's more useful information I can capture, if I see it again.

Just happened again - same signature described above, this time transitioning between 1000 -> 2000 concurrent requests.

<img width="1616" height="549" alt="Image" src="https://github.com/user-attachments/assets/6d7fc96a-f094-42b4-9bf8-8b83e907dc20" />

Using the guidellm development branch on python3.13 I can no longer reproduce the issue, looks like its been resolved now. 🚀