Repository: UKGovernmentBEIS/inspect_ai
Issue: Failed epoch doesn't stop rest of task's epochs
Root cause: Resource Mishandling

When running multiple tasks with multiple epochs, and a single epoch of one task fails (e.g. unhandled exception in solver), the console output suggests they are not running (red X, 100%, timer stopped).
However, it appears that other epochs of the failed task continue to run (unexpected).
<img width="556" alt="Screenshot 2024-10-23 at 14 06 33" src="https://github.com/user-attachments/assets/218f3306-bb82-429f-aa14-45ebb3d48a5d">

Once any other running tasks complete, any epochs of failed tasks which still are running are stopped. And if any other epochs of the failed task do succeed, they are not written to the log, so it seems like wasted effort.

Here's a repro, which I've attached a screen recording of (see the containers on the right hand side) with `watch docker ps -q`

https://github.com/user-attachments/assets/33c2f4f9-fdf0-4057-abfa-eb1d5e40d235

```py
import asyncio
import logging
from inspect_ai import Task, task, eval
from inspect_ai.dataset import MemoryDataset, Sample

from inspect_ai.solver import Generate, TaskState, solver


logger = logging.getLogger(__name__)

# Observed behaviour:
# * 4 containers get created immediately (as expected)
# * task_fail_once epoch 1 fails (as expected)
# * The console output shows task_fail_once as failed, with 100% progress and timer
#   stopped:
# ╭─ eval: 1/2 tasks complete ─────────────────────────────────────────────────────────╮
# │ sleep_duration: 10, epochs: 2, max_samples: 4, max_tasks: 2                        │
# │                                                                                    │
# │ ⠿ task_sleep      mockllm/model  ━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━  27% 0:00:05 │
# │ ✗ task_fail_once  mockllm/model  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:01 │
# │                                                                                    │
# │ subprocesses: 0/16                                             HTTP rate limits: 0 │
# ╰────────────────────────────────────────────────────────────────────────────────────╯
# * task_fail_once epoch 1's container is deleted (as expected)
# * task_fail_once epoch 2's container sticks around, and solvers keep getting run (not
#   expected given console output)
# * once task_sleep completes, task_fail_once is stopped.

# Expected behaviour:
# Either the console output should convey that there is still some progress with
# task_fail_once, or all epochs for that task should be stopped.

def main():
    eval(
        tasks=[
            # This task just sleeps for 10s.
            task_sleep(10),
            # This task fails immediately for the first epochs, but sleeps for 20s on
            # the second.
            task_fail_once(20),
        ],
        model="mockllm/model",
        max_tasks=2,
        max_samples=4,
        epochs=2,
        log_level="info",
    )



@task
def task_sleep(sleep_duration: int):
    return Task(
        dataset=MemoryDataset([Sample(input="task_sleep", target="Target")]),
        sandbox="docker",
        solver=[sleep_solver()] * sleep_duration,
    )


@task
def task_fail_once(sleep_duration: int):
    return Task(
        dataset=MemoryDataset([Sample(input="task_fail_once", target="Target")]),
        sandbox="docker",
        solver=[fail_once_solver()] + [sleep_solver()] * sleep_duration,
    )


@solver
def sleep_solver():
    async def solve(state: TaskState, generate: Generate):
        await asyncio.sleep(1)
        logger.info(f"epoch {state.epoch} for task {state.input}")
        return state

    return solve

has_failed = False

@solver
def fail_once_solver():
    async def solve(state: TaskState, generate: Generate):
        global has_failed
        if not has_failed:
            has_failed = True
            logger.error("Deliberate failure")
            raise RuntimeError("This solver fails just once.")
        return state

    return solve


if __name__ == "__main__":
    main()
```

inspect_ai version `0.3.42.dev43+g1c13832e`

Happy to try out any potential fixes! Thanks a lot.

Looks like our move to anyio has resolved this!  (2 points for structured concurrency!). Here's what I now see:

```
(.venv) jjallaire@JJs-MacBook-Pro inspect_ai % python3 examples/repro.py
[+] Pulling 1/1
 ✔ default Pulled                                                                                            1.4s 

[04/20/25 17:51:47] ERROR    Deliberate failure                                                        repro.py:88
[04/20/25 17:51:48] INFO     epoch 2 for task task_sleep                                               repro.py:73
                    INFO     epoch 1 for task task_sleep                                               repro.py:73
[04/20/25 17:51:49] INFO     epoch 2 for task task_sleep                                               repro.py:73
                    INFO     epoch 1 for task task_sleep                                               repro.py:73
[04/20/25 17:51:50] INFO     epoch 2 for task task_sleep                                               repro.py:73
                    INFO     epoch 1 for task task_sleep                                               repro.py:73
[04/20/25 17:51:51] INFO     epoch 2 for task task_sleep                                               repro.py:73
                    INFO     epoch 1 for task task_sleep                                               repro.py:73
[04/20/25 17:51:52] INFO     epoch 2 for task task_sleep                                               repro.py:73
                    INFO     epoch 1 for task task_sleep                                               repro.py:73
[04/20/25 17:51:53] INFO     epoch 2 for task task_sleep                                               repro.py:73
                    INFO     epoch 1 for task task_sleep                                               repro.py:73
[04/20/25 17:51:54] INFO     epoch 2 for task task_sleep                                               repro.py:73
                    INFO     epoch 1 for task task_sleep                                               repro.py:73
[04/20/25 17:51:55] INFO     epoch 2 for task task_sleep                                               repro.py:73
                    INFO     epoch 1 for task task_sleep                                               repro.py:73
[04/20/25 17:51:56] INFO     epoch 2 for task task_sleep                                               repro.py:73
                    INFO     epoch 1 for task task_sleep                                               repro.py:73
[04/20/25 17:51:57] INFO     epoch 2 for task task_sleep                                               repro.py:73
[04/20/25 17:51:58] INFO     epoch 1 for task task_sleep                                               repro.py:73
╭─ task_sleep (1 x 2 samples): mockllm/model ────────────────────────────────────────────────────────────────────╮
│ epochs: 2, max_samples: 4, max_tasks: 2, sleep_duration: 10                                 dataset: (samples) │
│                                                                                                                │
│ total time:                                                    0:00:12                                         │
│                                                                                                                │
│                                                                                                                │
│ Log: logs/2025-04-20T17-51-46+01-00_task-sleep_PCqfXCTFBW6FRkoEycn3fm.eval                                     │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ task_fail_once (1 x 2 samples): mockllm/model ────────────────────────────────────────────────────────────────╮
│ ╭──────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────╮ │
│ │ /Users/jjallaire/ukinstitute/UKGovernmentBEIS/inspect_ai/src/inspect_ai/_util/_async.py:57 in tg_collect   │ │
│ │                                                                                                            │ │
│ │ /Users/jjallaire/ukinstitute/UKGovernmentBEIS/inspect_ai/.venv/lib/python3.12/site-packages/anyio/_backen… │ │
│ │ in __aexit__                                                                                               │ │
│ │                                                                                                            │ │
│ │    764 │   │   │   │                                                                                       │ │
│ │    765 │   │   │   │   self._active = False                                                                │ │
│ │    766 │   │   │   │   if self._exceptions:                                                                │ │
│ │ ❱  767 │   │   │   │   │   raise BaseExceptionGroup(                                                       │ │
│ │    768 │   │   │   │   │   │   "unhandled errors in a TaskGroup", self._exceptions                         │ │
│ │    769 │   │   │   │   │   )                                                                               │ │
│ │    770 │   │   │   │   elif exc_val:                                                                       │ │
│ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)                                              │
│                                                                                                                │
│ During handling of the above exception, another exception occurred:                                            │
│                                                                                                                │
│ ╭──────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────╮ │
│ │ /Users/jjallaire/ukinstitute/UKGovernmentBEIS/inspect_ai/src/inspect_ai/_eval/task/run.py:305 in task_run  │ │
│ │                                                                                                            │ │
│ │ /Users/jjallaire/ukinstitute/UKGovernmentBEIS/inspect_ai/src/inspect_ai/_util/_async.py:72 in tg_collect   │ │
│ │                                                                                                            │ │
│ │ /Users/jjallaire/ukinstitute/UKGovernmentBEIS/inspect_ai/src/inspect_ai/_util/_async.py:60 in run_task     │ │
│ │                                                                                                            │ │
│ │ /Users/jjallaire/ukinstitute/UKGovernmentBEIS/inspect_ai/src/inspect_ai/_eval/task/run.py:900 in           │ │
│ │ task_run_sample                                                                                            │ │
│ │                                                                                                            │ │
│ │ /Users/jjallaire/ukinstitute/UKGovernmentBEIS/inspect_ai/src/inspect_ai/_eval/task/run.py:679 in           │ │
│ │ task_run_sample                                                                                            │ │
│ │                                                                                                            │ │
│ │ /Users/jjallaire/ukinstitute/UKGovernmentBEIS/inspect_ai/src/inspect_ai/solver/_plan.py:106 in __call__    │ │
│ │                                                                                                            │ │
│ │ /Users/jjallaire/ukinstitute/UKGovernmentBEIS/inspect_ai/examples/repro.py:89 in solve                     │ │
│ │                                                                                                            │ │
│ │   86 │   │   if not has_failed:                                                                            │ │
│ │   87 │   │   │   has_failed = True                                                                         │ │
│ │   88 │   │   │   logger.error("Deliberate failure")                                                        │ │
│ │ ❱ 89 │   │   │   raise RuntimeError("This solver fails just once.")                                        │ │
│ │   90 │   │   return state                                                                                  │ │
│ │   91 │                                                                                                     │ │
│ │   92 │   return solve                                                                                      │ │
│ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ RuntimeError: This solver fails just once.                                                                     │
│                                                                                                                │
│ Task interrupted (no samples completed before interruption)                                                    │
│                                                                                                                │
│ Log: logs/2025-04-20T17-51-46+01-00_task-fail-once_hErSTpKFqAoBxWcH8pJweR.eval                                 │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────
```

Nice! Really glad to hear this. Just ran this myself again and agree.

The only thing I noted was that 3 containers stay up until the end of the tasks. Not a big concern from me at this stage, and I agree that this issue is closed in that we're not still running the solvers / making LLM calls for epochs which are bound to be discarded.

https://github.com/user-attachments/assets/3f117f4b-f8c7-4367-ab1b-c98635254ec6