Repository: vllm-project/guidellm
Issue: High concurrency benchmarks (rate >= 24) cause unhandled ConnectionRefusedError on shutdown
Root cause: Resource Mishandling

**Describe the bug**

When running a benchmark with rate-type="concurrent", a ConnectionRefusedError traceback is consistently printed at the end of the run if the concurrency rate is high (e.g., >= 24). The error does not occur at lower concurrency rates (e.g., <= 20).

Despite the error, the final benchmark statistics appear to be correct and complete. However, the presence of an unhandled exception indicates an unclean shutdown process.

**Expected behavior**

The benchmark should complete without any errors or exceptions printed to the console, regardless of the concurrency rate. The process shutdown should be graceful.

**Environment**
Include all relevant environment information:
1. OS: macOS (darwin)
2. Python version: 3.10.x

**To Reproduce**
Exact steps to reproduce the behavior:
The bug can be reliably reproduced by comparing a low-rate run with a high-rate run.
1. Run the "Good" Case (No Error): Execute the benchmark with rate=20. This command completes cleanly every time.

guidellm benchmark \
  --target "http://<your_model_endpoint>" \
  --processor "Qwen/Qwen3-0.6B" \
  --rate-type "concurrent" \
  --rate 20 \
  --max-requests 20 \
  --data "prompt_tokens=16,output_tokens=16,samples=1"

 2. Run the "Bad" Case (Error Occurs): Execute the benchmark with rate=24 or higher. This command consistently produces the error at the end of the run.

guidellm benchmark \
  --target "http://<your_model_endpoint>" \
  --processor "Qwen/Qwen3-0.6B" \
  --rate-type "concurrent" \
  --rate 24 \
  --max-requests 24 \
  --data "prompt_tokens=16,output_tokens=16,samples=1"

  Errors

  The following traceback is printed to the console on failure:

    1 Task exception was never retrieved
    2 future: <Task finished name='Task-22' coro=<RequestsWorker.send_result() done, defined at 
      /path/to/guidellm/src/guidellm/scheduler/worker.py:111> exception=ConnectionRefusedError(61, 'Connection refused')>
    3 Traceback (most recent call last):
    4   File "/path/to/python3.10/multiprocessing/managers.py", line 810, in _callmethod
    5     conn = self._tls.connection
    6 AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
    7 
    8 During handling of the above exception, another exception occurred:
    9 
   10 Traceback (most recent call last):
   11   File "/path/to/guidellm/src/guidellm/scheduler/worker.py", line 116, in send_result
   12     await asyncio.to_thread(results_queue.put, result)
   13   File "/path/to/python3.10/asyncio/threads.py", line 25, in to_thread
   14     return await loop.run_in_executor(None, func_call)
   15   File "/path/to/python3.10/concurrent/futures/thread.py", line 58, in run
   16     result = self.fn(*self.args, **self.kwargs)
   17   File "<string>", line 2, in put
   18   File "/path/to/python3.10/multiprocessing/managers.py", line 814, in _callmethod
   19     self._connect()
   20   File "/path/to/python3.10/multiprocessing/managers.py", line 801, in _connect
   21     conn = self._Client(self._token.address, authkey=self._authkey)
   22   File "/path/to/python3.10/multiprocessing/connection.py", line 502, in Client
   23     c = SocketClient(address)
   24   File "/path/to/python3.10/multiprocessing/connection.py", line 630, in SocketClient
   25     s.connect(address)
   26 ConnectionRefusedError: [Errno 61] Connection refused


**Errors**
If applicable, add a full print-out of any errors or exceptions that are raised or include screenshots to help explain your problem.

**Additional context**

Rate 20, tested 10 times, all normal.
At rates 22 and 23, errors began to appear.
At a rate of 24, the error rate is almost 100%.

guidellm benchmark \
  --target "http://10.64.24.34:8000" \
  --processor "Qwen/Qwen3-0.6B" \
  --rate-type "concurrent" \
  --rate 24 \
  --max-requests 24 \
  --data "prompt_tokens=16,output_tokens=16,samples=1"
Creating backend...
Backend openai_http connected to http://10.64.24.34:8000 for model qwen3-06b-model-2.
Creating request loader...
Created loader with 1 unique requests from prompt_tokens=16,output_tokens=16,samples=1.
                                                                                       
                                                                                       
╭─ Benchmarks ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ [14:57:58] ⠋   0% concurrent@24 (running )   Req:    0.0 req/s,    0.00s Lat,     0.0 Conc,       0 Comp,        0 Inc,        0 Err                       │
│                                              Tok:    0.0 gen/s,    0.0 tot/s,   0.0ms TTFT,    0.0ms ITL,     0 Prompt,        0 Gen                       │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Generating... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ (0/1) [ 0:00:05 < -:--:-- ]Task exception was never retrieved
future: <Task finished name='Task-22' coro=<RequestsWorker.send_result() done, defined at /Users/xx/trae-repository/guidellm/src/guidellm/scheduler/worker.py:111> exception=ConnectionRefusedError(61, 'Connection refused')>
Traceback (most recent call last):
  File "/Users/xx/.asdf/installs/python/3.10.17/lib/python3.10/multiprocessing/managers.py", line 810, in _callmethod
    conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/xx/trae-repository/guidellm/src/guidellm/scheduler/worker.py", line 116, in send_result
    await asyncio.to_thread(results_queue.put, result)  # type: ignore[attr-defined]
  File "/Users/xx/.asdf/installs/python/3.10.17/lib/python3.10/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
  File "/Users/xx/.asdf/installs/python/3.10.17/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "<string>", line 2, in put
  File "/Users/xx/.asdf/installs/python/3.10.17/lib/python3.10/multiprocessing/managers.py", line 814, in _callmethod
    self._connect()
  File "/Users/xx/.asdf/installs/python/3.10.17/lib/python3.10/multiprocessing/managers.py", line 801, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/Users/xx/.asdf/installs/python/3.10.17/lib/python3.10/multiprocessing/connection.py", line 502, in Client
    c = SocketClient(address)
  File "/Users/xx/.asdf/installs/python/3.10.17/lib/python3.10/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 61] Connection refused
Task exception was never retrieved
future: <Task finished name='Task-20' coro=<RequestsWorker.send_result() done, defined at /Users/xx/trae-repository/guidellm/src/guidellm/scheduler/worker.py:111> exception=ConnectionRefusedError(61, 'Connection refused')>
Traceback (most recent call last):
  File "/Users/xx/.asdf/installs/python/3.10.17/lib/python3.10/multiprocessing/managers.py", line 810, in _callmethod
    conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/xx/trae-repository/guidellm/src/guidellm/scheduler/worker.py", line 116, in send_result
    await asyncio.to_thread(results_queue.put, result)  # type: ignore[attr-defined]
  File "/Users/xx/.asdf/installs/python/3.10.17/lib/python3.10/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
  File "/Users/xx/.asdf/installs/python/3.10.17/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "<string>", line 2, in put
  File "/Users/xx/.asdf/installs/python/3.10.17/lib/python3.10/multiprocessing/managers.py", line 814, in _callmethod
    self._connect()
  File "/Users/xx/.asdf/installs/python/3.10.17/lib/python3.10/multiprocessing/managers.py", line 801, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/Users/xx/.asdf/installs/python/3.10.17/lib/python3.10/multiprocessing/connection.py", line 502, in Client
╭─ Benchmarks ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ [14:57:58] ⠴ 100% concurrent@24 (complete)   Req:  100.1 req/s,    0.18s Lat,    18.1 Conc,      24 Comp,        0 Inc,        0 Err                       │
│                                              Tok: 1598.0 gen/s, 4001.4 tot/s,  64.3ms TTFT,    7.7ms ITL,    24 Prompt,       16 Gen                       │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Generating... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ (1/1) [ 0:00:06 < 0:00:00 ]
                    
                    
Benchmarks Metadata:
    Run id:46316209-6412-4f31-8df9-9d6e00a40ff7
    Duration:7.1 seconds
    Profile:type=concurrent, strategies=['concurrent'], streams=[24]
    Args:max_number=24, max_duration=None, warmup_number=None, warmup_duration=None, cooldown_number=None, cooldown_duration=None
    Worker:type_='generative_requests_worker' backend_type='openai_http' backend_target='http://10.64.24.34:8000' backend_model='qwen3-06b-model-2'           
    backend_info={'max_output_tokens': 16384, 'timeout': 300, 'http2': True, 'follow_redirects': True, 'headers': {}, 'text_completions_path':                
    '/v1/completions', 'chat_completions_path': '/v1/chat/completions'}                                                                                       
    Request Loader:type_='generative_request_loader' data='prompt_tokens=16,output_tokens=16,samples=1' data_args=None processor='Qwen/Qwen3-0.6B'            
    processor_args=None                                                                                                                                       
    Extras:None
                
                
Benchmarks Info:
======================================================================================================================================================
Metadata                                      |||| Requests Made  ||| Prompt Tok/Req ||| Output Tok/Req ||| Prompt Tok Total  ||| Output Tok Total  ||
    Benchmark| Start Time| End Time| Duration (s)|  Comp|  Inc|  Err|  Comp|  Inc|  Err|  Comp|  Inc|  Err|   Comp|   Inc|   Err|   Comp|   Inc|   Err
-------------|-----------|---------|-------------|------|-----|-----|------|-----|-----|------|-----|-----|-------|------|------|-------|------|------
concurrent@24|   14:58:04| 14:58:05|          0.2|    24|    0|    0|  24.0|  0.0|  0.0|  16.0|  0.0|  0.0|    576|     0|     0|    384|     0|     0
======================================================================================================================================================
                 
                 
Benchmarks Stats:
==================================================================================================================================================
Metadata     | Request Stats         || Out Tok/sec| Tot Tok/sec| Req Latency (sec)  ||| TTFT (ms)        ||| ITL (ms)       ||| TPOT (ms)      ||
    Benchmark| Per Second| Concurrency|        mean|        mean|  mean|  median|   p99| mean| median|   p99| mean| median| p99| mean| median| p99
-------------|-----------|------------|------------|------------|------|--------|------|-----|-------|------|-----|-------|----|-----|-------|----
concurrent@24|     100.14|       18.06|      1598.0|      4001.4|  0.18|    0.16|  0.23| 64.3|   44.2| 118.3|  7.7|    7.7| 8.1|  7.2|    7.2| 7.6
==================================================================================================================================================
                           
Saving benchmarks report...
Benchmarks report saved to /Users/xx/trae-repository/guidellm/benchmarks.json
                      
Benchmarking complete.

This might be a MacOS specific issue. We're often testing at concurrences in the high 100s to 1000s and do not have this problem. Can you try setting `export GUIDELLM__MAX_WORKER_PROCESSES=10`?

Can you also list what version of GuideLLM your using?

> Can you also list what version of GuideLLM your using?

Even though I'm using the latest code from the main branch of guidellm, installed with pip install -e, and have updated to yesterday's version, I'm getting the same error

guidellm --version
guidellm version: 0.4.0.dev2

> 这可能是 MacOS 独有的问题。我们经常在并发数高达 100 到 1000 的情况下进行测试，并没有遇到这个问题。您可以尝试设置 `export GUIDELLM__MAX_WORKER_PROCESSES=10` 吗？

On my M1 Pro Mac with an 8-core CPU, I found that the default value for GUIDELLM__MAX_WORKER_PROCESSES was too low. When I increased it to 20, the rate could stabilize at 50, but anything higher would cause an error. When I changed it to 30, it could stabilize at 80, but again, anything higher would lead to the same error. Beyond a rate of 80, increasing GUIDELLM__MAX_WORKER_PROCESSES further had no effect.
Additionally, this seems to be related to the CPU pressure caused by the token size. At the same rate, larger tokens frequently lead to failures, while smaller tokens are almost always successful

(.venv) xxx@MacBook-Pro guidellm % export GUIDELLM__MAX_WORKER_PROCESSES=30
(.venv) xxx@MacBook-Pro guidellm % guidellm benchmark \
  --target "http://10.64.24.34:8000" \
  --processor "Qwen/Qwen3-0.6B" \
  --rate-type "concurrent" \
  --rate 80 \
  --max-seconds 20 \
  --data "prompt_tokens=256,output_tokens=256,samples=1"
Creating backend...
Backend openai_http connected to http://10.64.24.34:8000 for model Qwen/Qwen3-0.6B.
Creating request loader...
Created loader with 1 unique requests from prompt_tokens=256,output_tokens=256,samples=1.
                                                                                         
                                                                                         
╭─ Benchmarks ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ [14:34:45] ⠴ 100% concurrent@80 (running )   Req:    9.2 req/s,    5.51s Lat,    80.0 Conc,     240 Comp,        0 Inc,        0 Err                  │
│                                              Tok: 2356.6 gen/s, 4713.2 tot/s,  71.0ms TTFT,   21.3ms ITL,   256 Prompt,      256 Gen                  │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Generating... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ (0/1) [ 0:00:20 < 0:00:00 ]Task exception was never retrieved
future: <Task finished name='Task-175' coro=<RequestsWorker.send_result() done, defined at /Users/xxx/trae-repository/guidellm/src/guidellm/scheduler/worker.py:111> exception=ConnectionRefusedError(61, 'Connection refused')>
Traceback (most recent call last):
  File "/Users/xxx/.asdf/installs/python/3.10.17/lib/python3.10/multiprocessing/managers.py", line 810, in _callmethod
    conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/xxx/trae-repository/guidellm/src/guidellm/scheduler/worker.py", line 116, in send_result
    await asyncio.to_thread(results_queue.put, result)  # type: ignore[attr-defined]
  File "/Users/xxx/.asdf/installs/python/3.10.17/lib/python3.10/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
  File "/Users/xxx/.asdf/installs/python/3.10.17/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "<string>", line 2, in put
  File "/Users/xxx/.asdf/installs/python/3.10.17/lib/python3.10/multiprocessing/managers.py", line 814, in _callmethod
    self._connect()
  File "/Users/xxx/.asdf/installs/python/3.10.17/lib/python3.10/multiprocessing/managers.py", line 801, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/Users/xxx/.asdf/installs/python/3.10.17/lib/python3.10/multiprocessing/connection.py", line 502, in Client
    c = SocketClient(address)
  File "/Users/xxx/.asdf/installs/python/3.10.17/lib/python3.10/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 61] Connection refused

(.venv) xxx@MacBook-Pro guidellm % guidellm benchmark \
  --target "http://10.64.24.34:8000" \
  --processor "Qwen/Qwen3-0.6B" \
  --rate-type "concurrent" \
  --rate 80 \
  --max-seconds 20 \
  --data "prompt_tokens=8,output_tokens=8,samples=1"
Creating backend...
Backend openai_http connected to http://10.64.24.34:8000 for model Qwen/Qwen3-0.6B.
Creating request loader...
Created loader with 1 unique requests from prompt_tokens=8,output_tokens=8,samples=1.
                                                                                     
                                                                                     
╭─ Benchmarks ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ [14:35:42] ⠼ 100% concurrent@80 (complete)   Req:  445.0 req/s,    0.17s Lat,    74.4 Conc,    8008 Comp,       78 Inc,        0 Err                  │
│                                              Tok: 3559.9 gen/s, 7119.8 tot/s,  62.5ms TTFT,   14.9ms ITL,     8 Prompt,        8 Gen                  │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Generating... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ (1/1) [ 0:00:20 < 0:00:00 ]
                    
                    
Benchmarks Metadata:
    Run id:a63c5988-0d40-4d9e-bfde-33fb31bbbbad
    Duration:28.2 seconds
    Profile:type=concurrent, strategies=['concurrent'], streams=[80]
    Args:max_number=None, max_duration=20.0, warmup_number=None, warmup_duration=None, cooldown_number=None, cooldown_duration=None
    Worker:type_='generative_requests_worker' backend_type='openai_http' backend_target='http://10.64.24.34:8000' backend_model='Qwen/Qwen3-0.6B'        
    backend_info={'max_output_tokens': 16384, 'timeout': 300, 'http2': True, 'follow_redirects': True, 'headers': {}, 'text_completions_path':           
    '/v1/completions', 'chat_completions_path': '/v1/chat/completions'}                                                                                  
    Request Loader:type_='generative_request_loader' data='prompt_tokens=8,output_tokens=8,samples=1' data_args=None processor='Qwen/Qwen3-0.6B'         
    processor_args=None                                                                                                                                  
    Extras:None
                
                
Benchmarks Info:
==================================================================================================================================================
Metadata                                      |||| Requests Made  ||| Prompt Tok/Req ||| Output Tok/Req ||| Prompt Tok Total||| Output Tok Total||
    Benchmark| Start Time| End Time| Duration (s)|  Comp|  Inc|  Err|  Comp|  Inc|  Err|  Comp|  Inc|  Err|   Comp|  Inc|  Err|   Comp|  Inc|  Err
-------------|-----------|---------|-------------|------|-----|-----|------|-----|-----|------|-----|-----|-------|-----|-----|-------|-----|-----
concurrent@80|   14:35:51| 14:36:09|         18.0|  8008|   78|    0|   8.0|  8.0|  0.0|   8.0|  5.6|  0.0|  64064|  624|    0|  64064|  439|    0
==================================================================================================================================================
                 
                 
Benchmarks Stats:
====================================================================================================================================================
Metadata     | Request Stats         || Out Tok/sec| Tot Tok/sec| Req Latency (sec)  ||| TTFT (ms)        ||| ITL (ms)        ||| TPOT (ms)       ||
    Benchmark| Per Second| Concurrency|        mean|        mean|  mean|  median|   p99| mean| median|   p99| mean| median|  p99| mean| median|  p99
-------------|-----------|------------|------------|------------|------|--------|------|-----|-------|------|-----|-------|-----|-----|-------|-----
concurrent@80|     444.99|       74.35|      3559.9|      7119.8|  0.17|    0.17|  0.22| 62.5|   60.1| 107.8| 14.9|   15.3| 18.6| 13.1|   13.4| 16.3
====================================================================================================================================================
                           
Saving benchmarks report...
Benchmarks report saved to /Users/xxx/trae-repository/guidellm/benchmarks.json
                      
Benchmarking complete.

Two more things to test. Please try `python3.13`, I have seen a few different issues with multiprocessing in older python versions. If that fails try our next release tracking branch: `feature/refactor/draft-main` I am nearly certain that we have solved this issue in the branch but you may run into other bugs as its still a work in progress.

> Two more things to test. Please try `python3.13`, I have seen a few different issues with multiprocessing in older python versions. If that fails try our next release tracking branch: `feature/refactor/draft-main` I am nearly certain that we have solved this issue in the branch but you may run into other bugs as its still a work in progress.


"Effective, the issue was solved after I switched from Python 3.10 to Python 3.13. The default parameters allow for testing with --rate 100."

Noting on here as an action item for us to update the docs / release notes that Python 3.10 has these issues and to retest if they are still present with the 0.4

Closing out as this should have been addressed in 0.4.0 release as well as the new container image fixes. Please reopen if you still see it happening