Repository: UKGovernmentBEIS/inspect_ai
Issue: Docker compose build failures are fatal even with `--no-fail-on-error`
Root cause: Validation Gap

If a docker compose project used in one task fails to pass a (docker compose) health check, Inspect seems to terminate the whole run, even when running with --no-fail-on-error. When this flag is present I'd think that one task would fail but others should still be run.

```
RuntimeError: Failed to start docker services for /home/.../compose.yml: 
Network inspect-my_benchmark_default  Creating
Network inspect-my_benchmark_default  Created
Container inspect-my_benchmark-target-1  Creating
Container inspect-my_benchmark-agent-1  Creating
Container inspect-my_benchmark-agent-1  Created
Container inspect-my_benchmark-target-1  Created
Container inspect-my_benchmark-user-1  Creating
Container inspect-my_benchmark-user-1  Created
Container inspect-my_benchmark-target-1  Starting
Container inspect-my_benchmark-agent-1  Starting
Container inspect-my_benchmark-agent-1  Started
Container inspect-my_benchmark-target-1  Started
Container inspect-my_benchmark-target-1  Waiting
Container inspect-my_benchmark-target-1  Error
timeout waiting for dependencies
```

Backtrace:
```
~/venv/lib/python3.12/site-packages/inspect_ai/_eval/task/run.py:320 in task_run
~/venv/lib/python3.12/site-packages/inspect_ai/_eval/task/run.py:548 in task_run_sample
~/.local/share/uv/python/cpython-3.12.4-linux-x86_64-gnu/lib/python3.12/contextlib.py:210 in __aenter__
207 │   │   # they are only needed for recreation, which is not possible anymore
208 │   │   del self.args, self.kwds, self.func
209 │   │   try:
❱ 210 │   │   │   return await anext(self.gen)
211 │   │   except StopAsyncIteration:
212 │   │   │   raise RuntimeError("generator didn't yield") from None
213
~/venv/lib/python3.12/site-packages/inspect_ai/_eval/task/sandbox.py:83 in sandboxenv_context
~/venv/lib/python3.12/site-packages/inspect_ai/util/_sandbox/context.py:128 in init_sandbox_environments_sample
~/venv/lib/python3.12/site-packages/inspect_ai/util/_sandbox/docker/docker.py:181 in sample_init
~/venv/lib/python3.12/site-packages/inspect_ai/util/_sandbox/docker/docker.py:140 in sample_init
~/venv/lib/python3.12/site-packages/inspect_ai/util/_sandbox/docker/compose.py:39 in compose_up
```

@jjallaire-aisi replied to this when I posted in in Slack with an initial diagnosis:
> In this case the sample hasn't "started running" yet so we don't have ready access to the data structures required to treat it as a failed sample. I think we can figure out a way to do this but it won't be a quick fix (need to think though the layers/implications carefully). Could you post this as a GitHub issue?

I've been encountering a similar issue, afaict there are two places where behavior would need to be changed to fix this.

Currently, Inspect errors out if `compose up` fails, which due to `--wait` flag, will fail unless all services are up and healthy. This could be fixed by adding a check for the `no_fail_on_error` flag in line 35 below.

https://github.com/UKGovernmentBEIS/inspect_ai/blob/326285f7c0b3f93e127a7296da3b6256a60f1d6a/src/inspect_ai/util/_sandbox/docker/compose.py#L27-L37

There's also another issue here - currently the docker compose `--wait` flag is bugged in such a way that it does not distinguish between failures and successful exits, meaning that e.g. an initialization container intended to complete during startup will trigger an error. (See more [here](https://github.com/docker/compose/issues/10596)) A proper fix would likely involve ignoring the returned result of the compose command, and instead manually checking the status of each container to ensure that it is not unhealthy, and that it has not exited with a non-zero code.

There already seems to be code that does this, but its bugged in pretty much the same way, ignoring the `no_fail_on_error` flag, and treating successful exits as errors.

https://github.com/UKGovernmentBEIS/inspect_ai/blob/326285f7c0b3f93e127a7296da3b6256a60f1d6a/src/inspect_ai/util/_sandbox/docker/compose.py#L94-L109

The best approach may be to remove the failure case from `compose_up`, and revise `compose_check_running` to recognize the flag and treat code-0 exits as successful. The failure case in `compose_up` is redundant with the current usage, as it is only used in one location, and `compose_check_running` is called directly after.

https://github.com/UKGovernmentBEIS/inspect_ai/blob/326285f7c0b3f93e127a7296da3b6256a60f1d6a/src/inspect_ai/util/_sandbox/docker/docker.py#L138-L145

@collingray Thanks for digging into all of this! Looking more carefully I think there is a relatively simple solution that will work for both the `compose up` scenario as well as errors that occur in setup scripts. We currently have a sandbox context manager that does all of this work before executing a sample. I think if we simply move the context manager _into_ the sample try/except then our standard sample error handling (which implements `--fail-on-error`) will handle the rest.

I need to verify that it really is that simple as well as that there aren't unintended side effects from the relocation of the context manager (on inspection it looks like not but need to 100% sure!). Will report back soon here on what I find + hopefully a fix.

@jjallaire That's a much better solution than what I was suggesting, but I'm now realizing that the issue I was encountering was related but distinct from the original, and so still exists.

I've put up a pr with the fix for the issue I was encountering, and would appreciate any input on it:
https://github.com/UKGovernmentBEIS/inspect_ai/pull/1111