Repository: UKGovernmentBEIS/inspect_ai
Issue: Occasional RuntimeErrors from bash_session tool
Root cause: Resource Mishandling

I set up a docker-based sandbox to support the Inspect runtime dependencies for the bash session tool as per the [instructions in the docs](https://inspect.aisi.org.uk/tools-standard.html#sec-bash-session) and I ran an eval using custom agent (that's mostly just a wrapper on the new react agent). The bash_session tool seems to work most of the time, but in 2 of 12 samples it eventually raises an error like what's shown below (just the command changes in the 2nd error).

I saw this behavior at current head of main (e7af455c471c413f978b1df36e4ce213b9d2683f) with a Kali-linux based container.

In the log I see a tool call:
```
inspect-tool-support exec
{"jsonrpc": "2.0", "method": "bash_session", "id": 758, "params": {"command": "grep -R \"SEKAI{\" / 2>/dev/null", "restart": null, "session_name": "BashSession"}}
```

And a full response/traceback is:
```
┌───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────┐
│ /home/dev/git/inspect_ai/src/inspect_ai/_eval/task/run.py:658 in task_run_sample                                     │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/solver/_plan.py:106 in __call__                                              │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/agent/_as_solver.py:56 in solve                                              │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/agent/_react.py:226 in execute                                               │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/model/_call_tools.py:325 in execute_tools                                    │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/model/_call_tools.py:138 in call_tool_task                                   │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/model/_call_tools.py:132 in call_tool_task                                   │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/model/_call_tools.py:388 in call_tool                                        │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/tool/_tools/_bash_session.py:113 in execute                                  │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/tool/_tool_support_helpers.py:87 in exec_sandbox_rpc                         │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
RuntimeError: Sandbox.exec failure executing bash_session(command: grep -R "SEKAI{" / 2>/dev/null, restart: None, session_name: BashSession): Traceback (most recent call last):
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        pool_request.request
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpcore/_async/http11.py", line 136, in handle_async_request
    raise exc
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpcore/_async/http11.py", line 106, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpcore/_async/http11.py", line 177, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpcore/_async/http11.py", line 217, in _receive_event
    data = await self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self.READ_NUM_BYTES, timeout=timeout
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpcore/_backends/anyio.py", line 32, in read
    with map_exceptions(exc_map):
         ~~~~~~~~~~~~~~^^^^^^^^^
  File "/usr/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/inspect_tool_support/_util/_json_rpc_helpers.py", line 114, in do_post
    return await client.post(url, json=body, timeout=30)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpx/_client.py", line 1859, in post
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
    ...<13 lines>...
    )
    ^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpx/_client.py", line 1540, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpx/_client.py", line 1629, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
    )
    ^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpx/_client.py", line 1730, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
    with map_httpcore_exceptions():
         ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/inspect_tool_support/bin/inspect-tool-support", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/inspect_tool_support/_cli/main.py", line 32, in main
    asyncio.run(_exec())
    ~~~~~~~~~~~^^^^^^^^^
  File "/usr/lib/python3.13/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File "/usr/lib/python3.13/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/inspect_tool_support/_cli/main.py", line 52, in _exec
    await (
    ...<3 lines>...
    )(request_json_str)
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/inspect_tool_support/_cli/main.py", line 65, in _dispatch_remote_method
    return await json_rpc_http_call(_SERVER_URL, request_json_str)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/inspect_tool_support/_util/_json_rpc_helpers.py", line 97, in json_rpc_http_call
    (await _retrying_post(url, json.loads(request_json_str))).text
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/inspect_tool_support/_util/_json_rpc_helpers.py", line 116, in _retrying_post
    return await do_post()
           ^^^^^^^^^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
  File "/opt/inspect_tool_support/lib/python3.13/site-packages/tenacity/__init__.py", line 421, in exc_check
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x7f7591bace10 state=finished raised ReadTimeout>]
```

When I reran this, I saw some `ConnectionResetError`s as well:

```
{
  "jsonrpc": "2.0",
  "error": {
    "code": -32603,
    "message": "ConnectionResetError('Connection lost')",
    "data": "Traceback (most recent call last):
 File &quot;/opt/inspect_tool_support/lib/python3.13/site-packages/inspect_tool_support/_util/_json_rpc_helpers.py&quot;, line 66, in with_validated_rpc_method_params
 match await handler(validated_params):
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File &quot;/opt/inspect_tool_support/lib/python3.13/site-packages/inspect_tool_support/_remote_tools/_bash_session/json_rpc_methods.py&quot;, line 44, in _bash_session
 return await controller.execute_command(session_name, command)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File &quot;/opt/inspect_tool_support/lib/python3.13/site-packages/inspect_tool_support/_remote_tools/_bash_session/controller.py&quot;, line 20, in execute_command
 return await self.session_for_name(session_name).execute_command(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
command, timeout
 ^^^^^^^^^^^^^^^^
)
 ^
 File &quot;/opt/inspect_tool_support/lib/python3.13/site-packages/inspect_tool_support/_remote_tools/_bash_session/bash_session.py&quot;, line 26, in execute_command
return await self._process.execute_command(command, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File &quot;/opt/inspect_tool_support/lib/python3.13/site-packages/inspect_tool_support/_remote_tools/_bash_session/bash_process.py&quot;, line 57, in execute_command
 await self._process.stdin.drain()
 File &quot;/usr/lib/python3.13/asyncio/streams.py&quot;, line 386, in drain
 await self._protocol._drain_helper()
 File &quot;/usr/lib/python3.13/asyncio/streams.py&quot;, line 166, in _drain_helper
 raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
"
  },
  "id": 722
}
```

@epatey a few thoughts here:

1. My theory is that that is an extremely long running bash command (such that it blows through default timeouts). @andrew-aisi is that possible?

2. Given that, we could try both setting a larger read timeout on the Httpx client as well as implementing the following other changes (this is based on code Anthropic uses to make sure that long running reasoning operations don't timeout or disconnect): https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/model/_openai.py#L658

3. If all of that fails, we should probably try to figure out a way to turn this into a either a tool timeout or at least a `ToolError` so the model gets feedback (ironically this also might be resolved by some of the other work we are doing on MCP!). 


It seems to happen on long running commands - above timeout but not particularly long (i.e., `grep`ing the full container's filesystem). I tried swapping some errors out for ToolErrors and it didn't immediately resolve the issue I was having, but there's a good chance my patch was flawed. I'll keep hacking on it and let you know if I get anywhere. Thanks for the quick reply!

Should be resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1600