Repository: UKGovernmentBEIS/inspect_ai
Issue: [BUG ?] [3.93] [MacOS] "Updated vLLM provider to use local server rather than in process vllm package (improved concurrency and resource utilization)."
Root cause: Environment Incompatibility

Hi !

Attempting to run a benchmark from inspect_evals with the latest (3.93) version of inspect_ai, I run into an exception.

```
inspect eval inspect_evals/mmlu_0_shot --model vllm/meta-llama/Llama-3.1-8B --log-level debug -M max_model_len=8192

[05/05/25 16:31:08] WARNING  Existing vLLM server not found. Starting new server for meta-llama/Llama-3.1-8B.                                                                                                        vllm.py:95
[05/05/25 16:31:09] DEBUG    No plugins for group vllm.platform_plugins found.                                                                                                                                   __init__.py:28
                    DEBUG    Checking if TPU platform is available.                                                                                                                                              __init__.py:34
                    DEBUG    TPU platform is not available because: No module named 'libtpu'                                                                                                                     __init__.py:44
                    DEBUG    Checking if CUDA platform is available.                                                                                                                                             __init__.py:52
                    DEBUG    Exception happens when checking CUDA platform: NVML Shared Library Not Found                                                                                                        __init__.py:76
                    DEBUG    CUDA platform is not available because: NVML Shared Library Not Found                                                                                                               __init__.py:93
                    DEBUG    Checking if ROCm platform is available.                                                                                                                                            __init__.py:100
                    DEBUG    ROCm platform is not available because: No module named 'amdsmi'                                                                                                                   __init__.py:114
                    DEBUG    Checking if HPU platform is available.                                                                                                                                             __init__.py:122
                    DEBUG    HPU platform is not available because habana_frameworks is not found.                                                                                                              __init__.py:129
                    DEBUG    Checking if XPU platform is available.                                                                                                                                             __init__.py:140
                    DEBUG    XPU platform is not available because: No module named 'intel_extension_for_pytorch'                                                                                               __init__.py:150
                    DEBUG    Checking if CPU platform is available.                                                                                                                                             __init__.py:158
                    DEBUG    Confirmed CPU platform is available because the machine is MacOS.                                                                                                                  __init__.py:168
                    DEBUG    Checking if Neuron platform is available.                                                                                                                                          __init__.py:180
                    DEBUG    Neuron platform is not available because: No module named 'transformers_neuronx'                                                                                                   __init__.py:187
                    DEBUG    Checking if CPU platform is available.                                                                                                                                             __init__.py:158
                    DEBUG    Confirmed CPU platform is available because the machine is MacOS.                                                                                                                  __init__.py:168
                    INFO     Automatically detected platform cpu.                                                                                                                                               __init__.py:239
                    INFO     Launching server on port 37623                                                                                                                                                 local_server.py:169
                    INFO     Started server with command: vllm serve meta-llama/Llama-3.1-8B --host 0.0.0.0 --api-key inspectai --max-model-len 8192 --port 37623                                           local_server.py:117
                    INFO     Waiting for server at http://localhost:37623 to become ready...                                                                                                                local_server.py:212
                    DEBUG    connect_tcp.started host='localhost' port=37623 local_address=None timeout=5.0 socket_options=None                                                                                    _trace.py:47
[05/05/25 16:31:11] INFO     Traceback (most recent call last):                                                                                                                                             local_server.py:108
                    INFO     File "/Users/user/miniforge3/envs/inspect/bin/vllm", line 8, in <module>                                                                                                   local_server.py:108
                    INFO     sys.exit(main())                                                                                                                                                               local_server.py:108
                    INFO     ^^^^^^                                                                                                                                                                         local_server.py:108
                    INFO     File "/Users/user/miniforge3/envs/inspect/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 51, in main                                                     local_server.py:108
                    INFO     args.dispatch_function(args)                                                                                                                                                   local_server.py:108
                    INFO     File "/Users/user/miniforge3/envs/inspect/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 27, in cmd                                                     local_server.py:108
                    INFO     uvloop.run(run_server(args))                                                                                                                                                   local_server.py:108
                    INFO     File "/Users/user/miniforge3/envs/inspect/lib/python3.12/site-packages/uvloop/__init__.py", line 109, in run                                                               local_server.py:108
                    INFO     return __asyncio.run(                                                                                                                                                          local_server.py:108
                    INFO     ^^^^^^^^^^^^^^                                                                                                                                                                 local_server.py:108
                    INFO     File "/Users/user/miniforge3/envs/inspect/lib/python3.12/asyncio/runners.py", line 195, in run                                                                             local_server.py:108
                    INFO     return runner.run(main)                                                                                                                                                        local_server.py:108
                    INFO     ^^^^^^^^^^^^^^^^                                                                                                                                                               local_server.py:108
                    INFO     File "/Users/user/miniforge3/envs/inspect/lib/python3.12/asyncio/runners.py", line 118, in run                                                                             local_server.py:108
                    INFO     return self._loop.run_until_complete(task)                                                                                                                                     local_server.py:108
                    INFO     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                            local_server.py:108
                    INFO     File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete                                                                                                      local_server.py:108
                    INFO     File "/Users/user/miniforge3/envs/inspect/lib/python3.12/site-packages/uvloop/__init__.py", line 61, in wrapper                                                            local_server.py:108
                    INFO     return await main                                                                                                                                                              local_server.py:108
                    INFO     ^^^^^^^^^^                                                                                                                                                                     local_server.py:108
                    INFO     File "/Users/user/miniforge3/envs/inspect/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1057, in run_server                                    local_server.py:108
                    INFO     sock = create_server_socket(sock_addr)                                                                                                                                         local_server.py:108
                    INFO     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                local_server.py:108
                    INFO     File "/Users/user/miniforge3/envs/inspect/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1028, in create_server_socket                          local_server.py:108
                    INFO     sock.bind(addr)                                                                                                                                                                local_server.py:108
                    INFO     OSError: [Errno 48] Address already in use                                                                                                                                     local_server.py:108
[05/05/25 16:31:14] DEBUG    connect_tcp.failed exception=ConnectError(ConnectionRefusedError(61, 'Connection refused'))                                                                                           _trace.py:47
                    DEBUG    Server check failed: [Errno 61] Connection refused, retrying...                                                                                                                local_server.py:251
[05/05/25 16:31:15] ERROR    Server process exited unexpectedly with code 1. Try rerunning with '--log-level debug' to see the full traceback. Alternatively, you can run the following launch command      local_server.py:232
                             manually to see the full traceback:

                             vllm serve meta-llama/Llama-3.1-8B --host 0.0.0.0 --api-key inspectai --max-model-len 8192 --port 37623

kill: 78172: No such process
```

while running :

```
vllm serve meta-llama/Llama-3.1-8B --host 0.0.0.0 --api-key inspectai --max-model-len 8192 --port 37623
```

works just fine.

Am I missing something here ?

Thanks !

@kaifronsdal Could you take a look at this?

I'm not entirely sure why this is happening. I tried to reproduce but it requires building vllm from source to run on a mac cpu. Perhaps we should change how we get a free port? We currently aren't closing it but relying on SO_REUSEADDR to make it so vllm can use it but perhaps there is a compatibility issue with mac. Another option would be to let vllm itself try to bind to a free port, and then we try to figure out somehow what the ports it has binded too aftrewards.

This old post does seem to point to why there is a discrepancy between MacOS and linux relationships to SO_REUSEADDR:

https://stackoverflow.com/questions/51998042/macos-so-reuseaddr-so-reuseport-not-consistent-with-linux 

@clebreto Would you be willing to take a shot at a PR to improve our port binding on MacOS?

> [@clebreto](https://github.com/clebreto) Would you be willing to take a shot at a PR to improve our port binding on MacOS?

Sounds like something I could do. 

Will look into this next week.

#1852