Repository: UKGovernmentBEIS/inspect_ai
Issue: Sandbox environments to resolve don't follow limit and sample-id filtering
Root cause: Algorithmic Error

One can limit the samples to eval with `--limit` and `--sample-id` options (applied [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/ade82ee61d5eb675cb24d92487349afac2447b62/src/inspect_ai/_eval/task/run.py#L161)). However, a sandbox is resolved for all samples in a dataset regardless of these options: the resolution code [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/ade82ee61d5eb675cb24d92487349afac2447b62/src/inspect_ai/_eval/run.py#L350) is called from `eval_run()` [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/ade82ee61d5eb675cb24d92487349afac2447b62/src/inspect_ai/_eval/run.py#L73C7-L79C10) before the filtering is applied.

I ran into this, e.g., when running the [cybench dataset](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/cybench) evals where resolving all the 40 sandboxes takes a lot of time to eval a single sample, and blocks running the evals if resolving a simple sandbox fails. Here's a snippet of log when running `inspect eval cybench.py@cybench --sample-id diffecient-hard --log-level DEBUG` - you can see it resolves unrelated sandboxes (ezmaze).

```
...
                    TRACE    Subprocess: docker version --format json (enter)                                                                                                                                                                       trace.py:41
                    TRACE    Subprocess: docker version --format json (exit)                                                                                                                                                                        trace.py:57
                    TRACE    Subprocess: docker compose version --format json (enter)                                                                                                                                                               trace.py:41
                    TRACE    Subprocess: docker compose version --format json (exit)                                                                                                                                                                trace.py:57
                    TRACE    Subprocess: docker compose --project-name inspect-startup-ii6skzy4nkpaojhqfxdbegh -f /Users/jan/dev/cybench/inspect_evals/src/inspect_evals/cybench/challenges/ezmaze/compose.yaml build (enter)                       trace.py:41
                    TRACE    Subprocess: docker compose --project-name inspect-startup-ii6skzy4nkpaojhqfxdbegh -f /Users/jan/dev/cybench/inspect_evals/src/inspect_evals/cybench/challenges/ezmaze/compose.yaml build (exit)                        trace.py:57
                    TRACE    Subprocess: docker compose --project-name inspect-startup-ii6skzy4nkpaojhqfxdbegh -f /Users/jan/dev/cybench/inspect_evals/src/inspect_evals/cybench/challenges/ezmaze/compose.yaml config --images (enter)             trace.py:41
                    TRACE    Subprocess: docker compose --project-name inspect-startup-ii6skzy4nkpaojhqfxdbegh -f /Users/jan/dev/cybench/inspect_evals/src/inspect_evals/cybench/challenges/ezmaze/compose.yaml config --images (exit)              trace.py:57
                    TRACE    Subprocess: docker compose --project-name inspect-startup-ii6skzy4nkpaojhqfxdbegh -f /Users/jan/dev/cybench/inspect_evals/src/inspect_evals/cybench/challenges/ezmaze/compose.yaml config (enter)                      trace.py:41
                    TRACE    Subprocess: docker compose --project-name inspect-startup-ii6skzy4nkpaojhqfxdbegh -f /Users/jan/dev/cybench/inspect_evals/src/inspect_evals/cybench/challenges/ezmaze/compose.yaml config (exit)                       trace.py:57

[01/05/25 17:14:40] DEBUG    open file: /Users/jan/dev/cybench/inspect_evals/src/inspect_evals/cybench/challenges/diffecient/images/victim/app/diffecient.py                                                                                       local.py:351
                    TRACE    Subprocess: docker compose --project-name inspect-cybench-ic5m2cz3ilzw8svpho6xcpe -f /Users/jan/dev/cybench/inspect_evals/src/inspect_evals/cybench/challenges/diffecient/compose.yaml config (enter)                  trace.py:41
                    TRACE    Subprocess: docker compose --project-name inspect-cybench-ic5m2cz3ilzw8svpho6xcpe -f /Users/jan/dev/cybench/inspect_evals/src/inspect_evals/cybench/challenges/diffecient/compose.yaml config (exit)                   trace.py:57
                    TRACE    Subprocess: docker compose --project-name inspect-cybench-ic5m2cz3ilzw8svpho6xcpe -f /Users/jan/dev/cybench/inspect_evals/src/inspect_evals/cybench/challenges/diffecient/compose.yaml up --detach --wait
...
```

I'm happy to contribute a PR for this if we agree on how to fix it. One may either refactor things more and do the filtering earlier (but then more places needs to be fixed, such as counting the total number of samples [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/ade82ee61d5eb675cb24d92487349afac2447b62/src/inspect_ai/_eval/task/run.py#L150) or numbering [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/ade82ee61d5eb675cb24d92487349afac2447b62/src/inspect_ai/_eval/run.py#L152)), or make a smaller change where we just duplicate the filtering logic from resolve_dataset() for the sandboxes.

Thanks for reporting this! This is a pretty unpleasant behavior which I agree we should remedy. I think we have what we need for a reusable way to apply these filters in the `slice_dataset()` function. I am not 100% sure what the best way to ensure this is applied to sandbox enumeration, let me take a closer look and I'll either post a PR or provide a suggested direction for you.

PR here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1079

Resolved here https://github.com/UKGovernmentBEIS/inspect_ai/pull/1079

Wonderful, thank you for the quick response!