Repository: UKGovernmentBEIS/inspect_ai
Issue: Web browser tool hangs indefinitely when running inspect_evals/gaia
Root cause: Environment Incompatibility

When running the inspect_evals/gaia task, the agent calls `web_browser_go()` and waits forever for a response.
I've waited 10+ minutes, so the timeout doesn't seem to work in this circumstance.

This happens consistently on any sample that involves a `web_broswer_go` call.
Clicking "Timeout Tool" makes the agent retry the tool call, but it also waits forever.

<img width="1163" alt="Image" src="https://github.com/user-attachments/assets/5b98f49e-8a0e-4195-b275-93a5468d4a34" />

Command:
`inspect eval inspect_evals/gaia --sample-id=c61d22de-5f6c-4958-a7f6-5e9707bd3466 --model=openai/gpt-4o`

`inspect trace anomalies` shows:
```
Tool Call    177.58s   web_browser_go(url='https://arxiv.org/')                                                                                 10:33:48 AEDT          
Subprocess   177.52s   docker compose --project-name inspect-gaia-itvzmtj -f                                                                   10:33:48 AEDT          
                      .../inspect_evals/src/inspect_evals/gaia/compose.yaml exec --workdir /app/web_browser                              
                      default python3 /app/web_browser/web_client_new_session.py
```

Logs from Docker container `inspect-gaia-iatme6e`:

This is logged on startup, or possibly on the first tool call:
```
Traceback (most recent call last):
  File "/app/web_browser/web_server.py", line 53, in <module>
    main()
  File "/app/web_browser/web_server.py", line 46, in main
    connection.send(dm_env_rpc_pb2.CreateWorldRequest())
  File "/usr/local/lib/python3.12/site-packages/dm_env_rpc/v1/connection.py", line 134, in send
    return message_utils.unpack_environment_response(self._stream.read(),
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/dm_env_rpc/v1/message_utils.py", line 208, in unpack_environment_response
    raise error.DmEnvRpcError(environment_response.error)
dm_env_rpc.v1.error.DmEnvRpcError: code: 13
message: "Browser.new_context: Target page, context or browser has been closed"
```
If you click "Timeout Tool" in the console, the following is logged:
```
Exception in thread Thread-1 (_serve):
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.12/site-packages/grpc/_server.py", line 1334, in _serve
    if not _process_event_and_continue(state, event):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/grpc/_server.py", line 1288, in _process_event_and_continue
    rpc_state, rpc_future = _handle_call(
                            ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/grpc/_server.py", line 1127, in _handle_call
    _handle_with_method_handler(
  File "/usr/local/lib/python3.12/site-packages/grpc/_server.py", line 1051, in _handle_with_method_handler
    return _handle_stream_stream(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/grpc/_server.py", line 975, in _handle_stream_stream
    return thread_pool.submit(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 173, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown
```

Architecture: Apple M4 Pro
OS: macOS Sequoia 15.2

The core problem appears to be that browsers can't be created in this environment (as the call failing here is in `web_client_new_session.py` which creates the browser). This could be a system or architecture specific issue (@craigwalton-dsit, @MattFisher note that they are on an M4 -- I wonder if we should consider publishing a multi-arch image here as well in case that's the issue).

We aren't able to reproduce this locally (on an M3) however we can address the timeout issue -- while the web browser's calls have a 3 minute timeout, the creation of a new session does not (so can indeed hang forever if the error is not properly reported). I will make the timeout change now but we still need to:

1) Figure out why the error doesn't propagate all the way up (we shouldn't be relying on a timeout when we have a hard error)
2) Figure out why this doesn't work on @MattFisher's configuration (perhaps M4 / arch related) 

Fix for timeout is here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/b7a0b38669f17fe700f0522312e9216dc8b97dc6

There is a multi-arch image for `aisiuk/inspect-web-browser-tool` with tag `multi-arch-testing` which you might want to try out. https://hub.docker.com/r/aisiuk/inspect-web-browser-tool/tags

The reason we haven't made the `latest` tag multi-arch yet is we wanted to get some mileage on the `inspect-computer-tool`'s multi arch image before making all images multi-arch.

Let us know if that makes a difference!

Okay, great! @MattFisher that basically implies that you change your image to point to:

```
aisiuk/inspect-web-browser-tool:multi-arch-testing
```

Let us know if that makes a difference!

It works when `inspect_evals/gaia/compose.yaml` points at `aisiuk/inspect-web-browser-tool:multi-arch-testing`!

For what it's worth, I did a bit of debugging in the `web_client_new_connection.py` file, and when I run it directly or duplicate the code in the python shell, it waits forever at line 27: `world_name = connection.send(dm_env_rpc_pb2.CreateWorldRequest()).world_name`

This is really useful feedback, thank you Matt.

@jjallaire in that case, should I build and publish the `inspect-web-browser-tool:latest` as multi-arch? This was the plan longer term anyway.

I've published a new `latest` image which is now multi-arch - hopefully this resolves this issue - thanks again for bring it to our attention.

https://hub.docker.com/r/aisiuk/inspect-web-browser-tool/tags

Thanks all for getting to the bottom of this!