Repository: UKGovernmentBEIS/inspect_ai
Issue: What conceptually does a "VLLM_API_KEY" mean, and how to integrate it into using vLLM on Huggingface models with Inspect?
Root cause: Environment Incompatibility

I'm quite concerned about how v0.3.93 of inspect seems to have broken our usual compatibility for vLLM inference of huggingface models! Namely, when we tried to infer one of our Huggingface models using vLLM on a colab notebook as shown below, we got an error of undefined VLLM_API_KEY:

![Image](https://github.com/user-attachments/assets/a688f1de-f34a-468e-95f3-5cf7c6493512)

But I don't even know what a "vllm_api_key" means conceptually -- neither vLLM nor huggingface has ever asked me for one in the past (and yes, I did load in my huggingface token as HF_TOKEN environment variable earlier). And when I went to the documentation at https://inspect.aisi.org.uk/providers.html#vllm for answers, they didn't seem to give us any examples or clarity, and neither did the changelog at https://github.com/UKGovernmentBEIS/inspect_ai/blob/bb714c10e046be7f0a1fea877d55aeea1e1b3e2d/CHANGELOG.md#L4. What should I do?

@kaifronsdal What is the right resolution of this? Is the vllm server looking for a key defined and it's not? (in which case should our client be defining a dummy key?).

The `VLLM_API_KEY` error is unrelated to why your script isn't working and just a byproduct of how we check to see if an existing server is running before creating our own. At the bottom of the error message there is a sugested command to run, can you try running it and see what happens? Alternatively try rerunning with log_level at debug.

#1806 Hopefully this will make the error source clearer. Unfortunately it is not trivial to tell users the exact cause of the server error ourselves since the server is being run in a seperate process. It is hard to filter through what in the output of the process is normal vllm/slgang debug/warning info and what is an error that should be propogated up to the user. This is why my current solution is to just tell users to to either run the start command themselves to see the output or to run inspect with everything being printed out (`--log-level=debug`)

Thanks @kaifronsdal! https://github.com/UKGovernmentBEIS/inspect_ai/pull/1806 is now merged.

@JoyeeChen let us know how you make out w/ the hopefully more clear error reporting.

When I ran the code with log_level = "debug" (but on a different model of around the same size), I got:
[05/05/25 18:07:53] WARNING  Existing vLLM server not found. Starting new server for                     vllm.py:92
                             unsloth/Llama-3.1-8B-Instruct.                                                        

[05/05/25 18:08:24] ERROR    Server process exited unexpectedly with code 1. Try rerunning with local_server.py:214
                             '--log-level debug' to see the full traceback. Alternatively, you                     
                             can run the following launch command manually to see the full                         
                             traceback:                                                                            
                                                                                                                   
                             vllm serve unsloth/Llama-3.1-8B-Instruct --host 0.0.0.0 --api-key                     
                             inspectai --tensor-parallel-size 1 --port 38446                                       
                                                                                                                   

---------------------------------------------------------------------------

PrerequisiteError                         Traceback (most recent call last)

/usr/local/lib/python3.11/dist-packages/inspect_ai/model/_providers/vllm.py in __init__(self, model_name, base_url, port, api_key, config, **server_args)
     80             # Try to initialize with existing server
---> 81             super().__init__(
     82                 model_name=model_name,

21 frames

/usr/local/lib/python3.11/dist-packages/inspect_ai/model/_providers/openai_compatible.py in __init__(self, model_name, base_url, api_key, config, service, service_base_url, **model_args)
     77             if not self.api_key:
---> 78                 raise environment_prerequisite_error(
     79                     self.service,

PrerequisiteError: ERROR: Unable to initialise vLLM client

No [bold][blue]VLLM_API_KEY[/blue][/bold] defined in the environment.


During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)

/usr/local/lib/python3.11/dist-packages/inspect_ai/_util/local_server.py in start_local_server(base_cmd, host, port, api_key, server_type, timeout, server_args)
    279         base_url = f"http://localhost:{found_port}/v1"
--> 280         wait_for_server(
    281             f"http://localhost:{found_port}",

/usr/local/lib/python3.11/dist-packages/inspect_ai/_util/local_server.py in wait_for_server(base_url, process, full_command, timeout, api_key)
    214             logger.error(error_msg)
--> 215             raise RuntimeError(error_msg)
    216 

RuntimeError: Server process exited unexpectedly with code 1. Try rerunning with '--log-level debug' to see the full traceback. Alternatively, you can run the following launch command manually to see the full traceback:

vllm serve unsloth/Llama-3.1-8B-Instruct --host 0.0.0.0 --api-key inspectai --tensor-parallel-size 1 --port 38446



The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)

<ipython-input-11-1b1e50c18923> in <cell line: 0>()
----> 1 log = eval(
      2     sample_task(),
      3     #model="vllm/CompassioninMachineLearning/10k_retrained_mixed_four_fifths_animals_march31_instruct_model_lr2e-5_3restarts",
      4     #0.54
      5     #model="vllm/CompassioninMachineLearning/10k_four_fifths_animals_PLORA_new",

/usr/local/lib/python3.11/dist-packages/inspect_ai/_eval/eval.py in eval(tasks, model, model_base_url, model_args, model_roles, task_args, sandbox, sandbox_cleanup, solver, tags, metadata, trace, display, approval, log_level, log_level_transcript, log_dir, log_format, limit, sample_id, epochs, fail_on_error, retry_on_error, debug_errors, message_limit, token_limit, time_limit, working_limit, max_samples, max_tasks, max_subprocesses, max_sandboxes, log_samples, log_realtime, log_images, log_buffer, log_shared, score, score_display, **kwargs)
    246                 raise
    247 
--> 248     return task_display().run_task_app(run_task_app)
    249 
    250 

/usr/local/lib/python3.11/dist-packages/inspect_ai/_display/rich/display.py in run_task_app(self, main)
     78     def run_task_app(self, main: Callable[[], Coroutine[None, None, TR]]) -> TR:
     79         if running_in_notebook():
---> 80             return run_coroutine(main())
     81         else:
     82             return anyio.run(main, backend=configured_async_backend())

/usr/local/lib/python3.11/dist-packages/inspect_ai/_util/_async.py in run_coroutine(coroutine)
    114     if running_in_notebook():
    115         init_nest_asyncio()
--> 116         return asyncio.run(coroutine)
    117     else:
    118         try:

/usr/local/lib/python3.11/dist-packages/nest_asyncio.py in run(main, debug)
     28         task = asyncio.ensure_future(main)
     29         try:
---> 30             return loop.run_until_complete(task)
     31         finally:
     32             if not task.done():

/usr/local/lib/python3.11/dist-packages/nest_asyncio.py in run_until_complete(self, future)
     96                 raise RuntimeError(
     97                     'Event loop stopped before Future completed.')
---> 98             return f.result()
     99 
    100     def _run_once(self):

/usr/lib/python3.11/asyncio/futures.py in result(self)
    201         self.__log_traceback = False
    202         if self._exception is not None:
--> 203             raise self._exception.with_traceback(self._exception_tb)
    204         return self._result
    205 

/usr/lib/python3.11/asyncio/tasks.py in __step(***failed resolving arguments***)
    275                 # We use the `send` method directly, because coroutines
    276                 # don't have `__iter__` and `__next__` methods.
--> 277                 result = coro.send(None)
    278             else:
    279                 result = coro.throw(exc)

/usr/local/lib/python3.11/dist-packages/inspect_ai/_eval/eval.py in run_task_app()
    199     async def run_task_app() -> list[EvalLog]:
    200         try:
--> 201             return await eval_async(
    202                 tasks=tasks,
    203                 model=model,

/usr/local/lib/python3.11/dist-packages/inspect_ai/_eval/eval.py in eval_async(tasks, model, model_base_url, model_args, model_roles, task_args, sandbox, sandbox_cleanup, solver, tags, metadata, approval, log_level, log_level_transcript, log_dir, log_format, limit, sample_id, epochs, fail_on_error, retry_on_error, debug_errors, message_limit, token_limit, time_limit, working_limit, max_samples, max_tasks, max_subprocesses, max_sandboxes, log_samples, log_realtime, log_images, log_buffer, log_shared, score, score_display, **kwargs)
    374     try:
    375         # intialise eval
--> 376         model = eval_init(
    377             model=model,
    378             model_base_url=model_base_url,

/usr/local/lib/python3.11/dist-packages/inspect_ai/_eval/eval.py in eval_init(model, model_base_url, model_args, max_subprocesses, log_level, log_level_transcript, **kwargs)
    932     # resolve and return models
    933     generate_config = GenerateConfig(**kwargs)
--> 934     models = resolve_models(model, model_base_url, model_args, generate_config)
    935     return models
    936 

/usr/local/lib/python3.11/dist-packages/inspect_ai/model/_model.py in resolve_models(model, model_base_url, model_args, config)
   1062 
   1063     # resolve models
-> 1064     return [resolve_model(m) for m in model]
   1065 
   1066 

/usr/local/lib/python3.11/dist-packages/inspect_ai/model/_model.py in <listcomp>(.0)
   1062 
   1063     # resolve models
-> 1064     return [resolve_model(m) for m in model]
   1065 
   1066 

/usr/local/lib/python3.11/dist-packages/inspect_ai/model/_model.py in resolve_model(m)
   1050     # helper to resolve model of various types
   1051     def resolve_model(m: str | Model) -> Model:
-> 1052         return get_model(
   1053             model=m,
   1054             base_url=model_base_url,

/usr/local/lib/python3.11/dist-packages/inspect_ai/model/_model.py in get_model(model, role, default, config, base_url, api_key, memoize, **model_args)
    998         init_hooks()
    999         modelapi_type = cast(type[ModelAPI], modelapi_types[0])
-> 1000         modelapi_instance = modelapi_type(
   1001             model_name=model,
   1002             base_url=base_url,

/usr/local/lib/python3.11/dist-packages/inspect_ai/model/_registry.py in model_wrapper(*args, **kwargs)
     54                 model_type = wrapped
     55 
---> 56             model = model_type(*args, **kwargs)
     57             registry_tag(
     58                 model_type,

/usr/local/lib/python3.11/dist-packages/inspect_ai/model/_providers/vllm.py in __init__(self, model_name, base_url, port, api_key, config, **server_args)
     99 
    100             # Start the server
--> 101             base_url, api_key = self._start_server(model_name, api_key=api_key)
    102             logger.warning(f"vLLM server started at {base_url}")
    103 

/usr/local/lib/python3.11/dist-packages/inspect_ai/model/_providers/vllm.py in _start_server(self, model_path, api_key)
    145         cmd = ["vllm", "serve", model_path, "--host", host, "--api-key", api_key]
    146 
--> 147         base_url, self.server_process, self.port = start_local_server(
    148             cmd,
    149             host=host,

/usr/local/lib/python3.11/dist-packages/inspect_ai/_util/local_server.py in start_local_server(base_cmd, host, port, api_key, server_type, timeout, server_args)
    292 
    293         # Re-raise with more context
--> 294         raise RuntimeError(f"Failed to start {server_type} server: {str(e)}") from e
    295 
    296 

RuntimeError: Failed to start vLLM server: Server process exited unexpectedly with code 1. Try rerunning with '--log-level debug' to see the full traceback. Alternatively, you can run the following launch command manually to see the full traceback:

vllm serve unsloth/Llama-3.1-8B-Instruct --host 0.0.0.0 --api-key inspectai --tensor-parallel-size 1 --port 38446



And when I ran, as directed, "!vllm serve unsloth/Llama-3.1-8B-Instruct --host 0.0.0.0 --api-key inspectai --tensor-parallel-size 1 --port 38446" on that same colab notebook (the ! is because it's inside a python cell), I got:
2025-05-05 18:52:48.426378: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1746471168.447429   14433 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1746471168.453786   14433 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
Traceback (most recent call last):
  File "/usr/local/bin/vllm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/cli/main.py", line 53, in main
    args.dispatch_function(args)
  File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/cli/serve.py", line 27, in cmd
    uvloop.run(run_server(args))
  File "/usr/local/lib/python3.11/dist-packages/uvloop/__init__.py", line 105, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.11/dist-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/api_server.py", line 1078, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/usr/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/usr/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/api_server.py", line 166, in build_async_engine_client_from_engine_args
    vllm_config = engine_args.create_engine_config(usage_context=usage_context)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/engine/arg_utils.py", line 1108, in create_engine_config
    if try_v1 and self._is_v1_supported_oracle(model_config):
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/engine/arg_utils.py", line 1352, in _is_v1_supported_oracle
    and current_platform.get_device_capability()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/platforms/cuda.py", line 61, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/platforms/cuda.py", line 328, in get_device_capability
    physical_device_id = device_id_to_physical_device_id(device_id)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/platforms/cuda.py", line 50, in device_id_to_physical_device_id
    return int(physical_device_id)
           ^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: 'None'

And because I suspect the problem might be related to whether or not vLLM recognizes the A100 instance google colab provides, I ran "os.environ["CUDA_VISIBLE_DEVICES"]" on the cell right after, and got output "None".

The CUDA_VISIBLE_DEVICES issue is known  #1803. Can you try and use the latest unreleased version of inspect (install from the git repo) or alternatively try setting devices=0 as a work around?

Update: I chose to first set os.environ["CUDA_VISIBLE_DEVICES"] = "0" near the top of my notebook, without using the latest unreleased version of inspect just yet.
Then, rerunning the cells, I got this output at the eval(...) line:
[05/05/25 19:24:58] WARNING  Existing vLLM server not found. Starting new server for                     vllm.py:92
                             unsloth/Llama-3.1-8B-Instruct.                                                        

DEBUG:pydot:pydot initializing

[05/05/25 19:25:00] DEBUG    pydot initializing                                                      __init__.py:15

DEBUG:pydot:pydot 3.0.4

                    DEBUG    pydot 3.0.4                                                             __init__.py:16

DEBUG:pydot.dot_parser:pydot dot_parser module initializing

                    DEBUG    pydot dot_parser module initializing                                  dot_parser.py:43

DEBUG:pydot.core:pydot core module initializing

                    DEBUG    pydot core module initializing                                              core.py:20

DEBUG:vllm.plugins:No plugins for group vllm.platform_plugins found.

[05/05/25 19:25:09] DEBUG    No plugins for group vllm.platform_plugins found.                       __init__.py:28

DEBUG:vllm.platforms:Checking if TPU platform is available.

                    DEBUG    Checking if TPU platform is available.                                  __init__.py:34

DEBUG:vllm.platforms:TPU platform is not available because: No module named 'libtpu'

                    DEBUG    TPU platform is not available because: No module named 'libtpu'         __init__.py:44

DEBUG:vllm.platforms:Checking if CUDA platform is available.

                    DEBUG    Checking if CUDA platform is available.                                 __init__.py:52

DEBUG:vllm.platforms:Confirmed CUDA platform is available.

                    DEBUG    Confirmed CUDA platform is available.                                   __init__.py:72

DEBUG:vllm.platforms:Checking if ROCm platform is available.

                    DEBUG    Checking if ROCm platform is available.                                __init__.py:100

DEBUG:vllm.platforms:ROCm platform is not available because: No module named 'amdsmi'

                    DEBUG    ROCm platform is not available because: No module named 'amdsmi'       __init__.py:114

DEBUG:vllm.platforms:Checking if HPU platform is available.

                    DEBUG    Checking if HPU platform is available.                                 __init__.py:122

DEBUG:vllm.platforms:HPU platform is not available because habana_frameworks is not found.

                    DEBUG    HPU platform is not available because habana_frameworks is not found.  __init__.py:129

DEBUG:vllm.platforms:Checking if XPU platform is available.

                    DEBUG    Checking if XPU platform is available.                                 __init__.py:140

DEBUG:vllm.platforms:XPU platform is not available because: No module named 'intel_extension_for_pytorch'

                    DEBUG    XPU platform is not available because: No module named                 __init__.py:150
                             'intel_extension_for_pytorch'                                                         

DEBUG:vllm.platforms:Checking if CPU platform is available.

                    DEBUG    Checking if CPU platform is available.                                 __init__.py:158

DEBUG:vllm.platforms:Checking if Neuron platform is available.

                    DEBUG    Checking if Neuron platform is available.                              __init__.py:180

DEBUG:vllm.platforms:Neuron platform is not available because: No module named 'transformers_neuronx'

                    DEBUG    Neuron platform is not available because: No module named              __init__.py:187
                             'transformers_neuronx'                                                                

DEBUG:vllm.platforms:Checking if CUDA platform is available.

                    DEBUG    Checking if CUDA platform is available.                                 __init__.py:52

DEBUG:vllm.platforms:Confirmed CUDA platform is available.

                    DEBUG    Confirmed CUDA platform is available.                                   __init__.py:72

INFO:vllm.platforms:Automatically detected platform cuda.

                    INFO     Automatically detected platform cuda.                                  __init__.py:239

DEBUG:tensorflow:Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.

[05/05/25 19:25:10] DEBUG    Falling back to TensorFlow client; we recommended you       tpu_cluster_resolver.py:34
                             install the Cloud TPU client directly with pip install                                
                             cloud-tpu-client.                                                                     

DEBUG:h5py._conv:Creating converter from 7 to 5

[05/05/25 19:25:11] DEBUG    Creating converter from 7 to 5                                          __init__.py:47

DEBUG:h5py._conv:Creating converter from 5 to 7

                    DEBUG    Creating converter from 5 to 7                                          __init__.py:47

DEBUG:h5py._conv:Creating converter from 7 to 5

                    DEBUG    Creating converter from 7 to 5                                          __init__.py:47

DEBUG:h5py._conv:Creating converter from 5 to 7

                    DEBUG    Creating converter from 5 to 7                                          __init__.py:47

DEBUG:2025-05-05 19:25:11,574:jax._src.path:31: etils.epath found. Using etils.epath for file I/O.
DEBUG:jax._src.path:etils.epath found. Using etils.epath for file I/O.

                    DEBUG    etils.epath found. Using etils.epath for file I/O.                          path.py:31

[05/05/25 19:25:13] INFO     Launching server on port 30344                                     local_server.py:156

                    INFO     Started server with command: vllm serve                            local_server.py:108
                             unsloth/Llama-3.1-8B-Instruct --host 0.0.0.0 --api-key inspectai                      
                             --tensor-parallel-size 1 --port 30344                                                 

                    INFO     Waiting for server at http://localhost:30344 to become ready...    local_server.py:197

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

                    DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:14] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:15] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:16] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:18] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

                    INFO     2025-05-05 19:25:18.985665: E                                       local_server.py:99
                             external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable                   
                             to register cuFFT factory: Attempting to register factory for                         
                             plugin cuFFT when one has already been registered                                     

[05/05/25 19:25:19] INFO     WARNING: All log messages before absl::InitializeLog() is called    local_server.py:99
                             are written to STDERR                                                                 

                    INFO     E0000 00:00:1746473119.008805    2486 cuda_dnn.cc:8310] Unable to   local_server.py:99
                             register cuDNN factory: Attempting to register factory for plugin                     
                             cuDNN when one has already been registered                                            

                    INFO     E0000 00:00:1746473119.015329    2486 cuda_blas.cc:1418] Unable to  local_server.py:99
                             register cuBLAS factory: Attempting to register factory for plugin                    
                             cuBLAS when one has already been registered                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

                    DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:20] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:21] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:22] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:23] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:24] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:25] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:26] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:27] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:28] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:29] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:30] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:31] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:32] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:33] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:35] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:36] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:37] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:38] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:39] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:40] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:41] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:42] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

[05/05/25 19:25:43] INFO     Traceback (most recent call last):                                  local_server.py:99

                    INFO     File "/usr/local/bin/vllm", line 8, in <module>                     local_server.py:99

                    INFO     sys.exit(main())                                                    local_server.py:99

                    INFO     ^^^^^^                                                              local_server.py:99

                    INFO     File                                                                local_server.py:99
                             "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/cli/main.                   
                             py", line 53, in main                                                                 

                    INFO     args.dispatch_function(args)                                        local_server.py:99

                    INFO     File                                                                local_server.py:99
                             "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/cli/serve                   
                             .py", line 27, in cmd                                                                 

                    INFO     uvloop.run(run_server(args))                                        local_server.py:99

                    INFO     File "/usr/local/lib/python3.11/dist-packages/uvloop/__init__.py",  local_server.py:99
                             line 105, in run                                                                      

                    INFO     return runner.run(wrapper())                                        local_server.py:99

                    INFO     ^^^^^^^^^^^^^^^^^^^^^                                               local_server.py:99

                    INFO     File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run     local_server.py:99

                    INFO     return self._loop.run_until_complete(task)                          local_server.py:99

                    INFO     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                 local_server.py:99

                    INFO     File "uvloop/loop.pyx", line 1518, in                               local_server.py:99
                             uvloop.loop.Loop.run_until_complete                                                   

                    INFO     File "/usr/local/lib/python3.11/dist-packages/uvloop/__init__.py",  local_server.py:99
                             line 61, in wrapper                                                                   

                    INFO     return await main                                                   local_server.py:99

                    INFO     ^^^^^^^^^^                                                          local_server.py:99

                    INFO     File                                                                local_server.py:99
                             "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/ap                   
                             i_server.py", line 1078, in run_server                                                

                    INFO     async with build_async_engine_client(args) as engine_client:        local_server.py:99

                    INFO     File "/usr/lib/python3.11/contextlib.py", line 210, in __aenter__   local_server.py:99

                    INFO     return await anext(self.gen)                                        local_server.py:99

                    INFO     ^^^^^^^^^^^^^^^^^^^^^                                               local_server.py:99

                    INFO     File                                                                local_server.py:99
                             "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/ap                   
                             i_server.py", line 146, in build_async_engine_client                                  

                    INFO     async with build_async_engine_client_from_engine_args(              local_server.py:99

                    INFO     File "/usr/lib/python3.11/contextlib.py", line 210, in __aenter__   local_server.py:99

                    INFO     return await anext(self.gen)                                        local_server.py:99

                    INFO     ^^^^^^^^^^^^^^^^^^^^^                                               local_server.py:99

                    INFO     File                                                                local_server.py:99
                             "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/ap                   
                             i_server.py", line 166, in                                                            
                             build_async_engine_client_from_engine_args                                            

                    INFO     vllm_config =                                                       local_server.py:99
                             engine_args.create_engine_config(usage_context=usage_context)                         

                    INFO     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^       local_server.py:99

                    INFO     File                                                                local_server.py:99
                             "/usr/local/lib/python3.11/dist-packages/vllm/engine/arg_utils.py",                   
                             line 1108, in create_engine_config                                                    

                    INFO     if try_v1 and self._is_v1_supported_oracle(model_config):           local_server.py:99

                    INFO     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                          local_server.py:99

                    INFO     File                                                                local_server.py:99
                             "/usr/local/lib/python3.11/dist-packages/vllm/engine/arg_utils.py",                   
                             line 1352, in _is_v1_supported_oracle                                                 

                    INFO     and current_platform.get_device_capability()                        local_server.py:99

                    INFO     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                            local_server.py:99

                    INFO     File                                                                local_server.py:99
                             "/usr/local/lib/python3.11/dist-packages/vllm/platforms/cuda.py",                     
                             line 61, in wrapper                                                                   

                    INFO     return fn(*args, **kwargs)                                          local_server.py:99

                    INFO     ^^^^^^^^^^^^^^^^^^^                                                 local_server.py:99

                    INFO     File                                                                local_server.py:99
                             "/usr/local/lib/python3.11/dist-packages/vllm/platforms/cuda.py",                     
                             line 328, in get_device_capability                                                    

                    INFO     physical_device_id = device_id_to_physical_device_id(device_id)     local_server.py:99

                    INFO     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                          local_server.py:99

                    INFO     File                                                                local_server.py:99
                             "/usr/local/lib/python3.11/dist-packages/vllm/platforms/cuda.py",                     
                             line 50, in device_id_to_physical_device_id                                           

                    INFO     return int(physical_device_id)                                      local_server.py:99

                    INFO     ^^^^^^^^^^^^^^^^^^^^^^^                                             local_server.py:99

                    INFO     ValueError: invalid literal for int() with base 10: 'None'          local_server.py:99

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

                    DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=30344 local_address=None timeout=5.0 socket_options=None

[05/05/25 19:25:44] DEBUG    connect_tcp.started host='localhost' port=30344 local_address=None        _trace.py:47
                             timeout=5.0 socket_options=None                                                       

DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign requested address'))

                    DEBUG    connect_tcp.failed exception=ConnectError(OSError(99, 'Cannot assign      _trace.py:47
                             requested address'))                                                                  

                    DEBUG    Server check failed: [Errno 99] Cannot assign requested address,   local_server.py:233
                             retrying...                                                                           

[05/05/25 19:25:45] ERROR    Server process exited unexpectedly with code 1. Try rerunning with local_server.py:214
                             '--log-level debug' to see the full traceback. Alternatively, you                     
                             can run the following launch command manually to see the full                         
                             traceback:                                                                            
                                                                                                                   
                             vllm serve unsloth/Llama-3.1-8B-Instruct --host 0.0.0.0 --api-key                     
                             inspectai --tensor-parallel-size 1 --port 30344                                       
                                                                                     

(that was followed by some error messages that looked sufficiently similar to the previous ones that I didn't include them here)

Then, when I ran 
`!vllm serve unsloth/Llama-3.1-8B-Instruct --host 0.0.0.0 --api-key inspectai --tensor-parallel-size 1 --port 30344`
on the next code block, I got this:
2025-05-05 19:26:31.022508: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1746473191.043694    2883 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1746473191.050140    2883 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
Traceback (most recent call last):
  File "/usr/local/bin/vllm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/cli/main.py", line 53, in main
    args.dispatch_function(args)
  File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/cli/serve.py", line 27, in cmd
    uvloop.run(run_server(args))
  File "/usr/local/lib/python3.11/dist-packages/uvloop/__init__.py", line 105, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.11/dist-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/api_server.py", line 1078, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/usr/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/usr/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/api_server.py", line 166, in build_async_engine_client_from_engine_args
    vllm_config = engine_args.create_engine_config(usage_context=usage_context)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/engine/arg_utils.py", line 1108, in create_engine_config
    if try_v1 and self._is_v1_supported_oracle(model_config):
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/engine/arg_utils.py", line 1352, in _is_v1_supported_oracle
    and current_platform.get_device_capability()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/platforms/cuda.py", line 61, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/platforms/cuda.py", line 328, in get_device_capability
    physical_device_id = device_id_to_physical_device_id(device_id)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/vllm/platforms/cuda.py", line 50, in device_id_to_physical_device_id
    return int(physical_device_id)
           ^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: 'None'

Update: I specified 
`os.environ["CUDA_VISIBLE_DEVICES"] = "0"`
and, within the eval(...) call, 
`model_args = {"device":"0"}`
and all went well. Thanks!


Update: I deleted that runtime, restarted a new colab runtime, then commented out the `os.environ["CUDA_VISIBLE_DEVICES"] = "0"` line (leaving only the ` model_args = {"device":"0"}` line) and it still worked!

@kaifronsdal Do we need to make any changes here to our default behavior as a result of the findings reported by @JoyeeChen ?

It should already be fixed by #1796 and https://github.com/UKGovernmentBEIS/inspect_ai/pull/1803!

I think the main potential improvement would be to find a way to either parse the output of the vllm/sglang servers to extract their error messages or to save a buffer of the server output that gets logged on server crash. This way it's easier to understand what went wrong and faster to debug issues with the local servers.