Repository: UKGovernmentBEIS/inspect_ai
Issue: docker provider write_file() hanging in `docker compose cp`
Root cause: Architectural Constraint

We have a reproducible case of an eval where a `sandbox().write_file()` in a setup solver will not complete due to `docker compose cp` hanging indefinitely. This is roughly the code (in a solver) that triggers this:

```python
await sandbox().write_file(file, content)
```

This is in an eval with 100 total samples (capped at 32 concurrent samples / sandboxes due to 16 logical cores). We are running on Ubuntu 24.04 on EC2 t3.2xlarge instance. Docker is installed from https://download.docker.com/linux/ubuntu as follows:

```bash
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
```

This is the rough form of the docker compose cp command that is hanging indefinitely:

```bash
docker compose --project-name inspect-task-inqf7ztt3dmpkmqhetves5x -f compose.yaml cp -- /tmp/tmp1z0bgjut content:/.tmp_inspect_sandbox_tmp1z0bgjut
```

We are able to resolve the issue by using a simpler form of writing files to the container (using `exec` with `tee`):

```python
await write_text_file("default", file, content)

async def write_text_file(sandbox_name: str, file: str, contents: str) -> None:
    result = await sandbox(sandbox_name).exec(["tee", "--", file], input=contents)
    if not result.success:
        msg = f"Failed to write file '{file}' into container: {result.stderr}"
        raise RuntimeError(msg) 
```

We have not yet been able to reproduce this on a Mac OS X machine with Docker. 

There are a few places we can go from here. I don't think we can reasonably expect that Docker on Linux will never hang on `compose cp` so we need to figure out ways to make this less common and less penal (hard to get more penal that an eval hanging forever with no explanation!):

1) Move away from the more complex `write_file()` implementation we now have (which is engineered to make sure that models calling `write_file()` aren't effectively writing with root privs). @art-dsit I believe you put this in for some AS evals that needed this behavior. Could this potentially be moved to "user space" (i.e. AS implements their own fancier write_file that does this) or perhaps become an option disabled by default on `write_file()`. I'm proposing here that we move away from using `docker compose cp` in any default codepath as this behavior is quite lethal!

2) Introduce a timeout for `write_file()` (enabled by default) that gives up after some reasonable time period e.g. 3 minutes. This would result in a hard error which would fail the eval (but it could still be retried successfully). In the case where `fail_on_error` is configured to allow some errors this would just fail the sample.

Certainly open to discussing any and all possible resolutions here!



We have investigated further and found that there may be a general problem with firing a ton of `docker compose` commands at the Docker daemon all at once. In the scenario above there is a call to `write_file()` in the first solver -- `write_file()` will in most cases call a combination of `docker compose cp` and `docker compose exec` 5 times, so during initialization of e.g. 30 concurrent samples a huge number of calls are made.

We have seen not only `docker compose cp` fail but also `docker compose exec` with a simple command (`id -g`). As discussed above, I think we may need to simplify `write_file()` to in most cases make a single call (and leave the fancier permission setting/validation for the application/tool layer). Note that `write_file()` is mostly used directly by solvers and not exposed to models (although it can be) so arguably the simpler, faster, and safer implementation better meets the normal use case.

In the meantime, there is now a 120 second timeout on `write_file()` which will at least prevent Docker from hanging (at the price of an occasional failed sample when a hang results in a timeout).

Resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1044