Repository: UKGovernmentBEIS/inspect_ai
Issue: High memory usage during eval-set retries
Root cause: Resource Mishandling

To reproduce:

Check out https://github.com/rasmusfaber/inspect_test_utils

Run:
```bash
uv sync
uv run inspect eval-set  --model hardcoded/hardcoded -M tool_calls="yes 1234567890ABCDEF | head -n 1000" -M repetitions=500 -T sample_count=100 -T failure_rate=0.5 --retry-wait 1 inspect_test_utils/sometimes_fails_scoring
```

Track the memory usage using `top` or another utility. Notice how the memory usage grows with ~0.5GB for each retry. Make sure to stop the evaluation before your machine gets sluggy.

I think I have tracked it down to the `PreviousTask`s [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/ec984508937238e3dd278ff879489b514498f99c/src/inspect_ai/_eval/evalset.py#L440-L447) not being released (they contain the full eval log), but I can't understand why they are not being released.

Tenacity is just calling the `try_eval()` function in a loop so that memory should definitely be freed. What happens if you force a gc in the `before()` function?

@jjallaire any chance your team could take over debugging this from the MRE provided?

> Tenacity is just calling the `try_eval()` function in a loop so that memory should definitely be freed. What happens if you force a gc in the `before()` function?

Basically same result. CPU works really hard, but only gets rid of some of the memory. The core problem that the memory usage grows with each retry persists (I assume that is unexpected right?)

Yes that is unexpected, especially since tenacity runs in a loop. I guess one problem with getting to the bottom of this is that we are calling a huge array of Python libraries, any one of which could be leaking or holding on to memory. What was the indication you had that `PreviousTask` was the culprit? Could you also give us an MRE that doesn't use internal METR stuff? 

I will also say that I am a bit surprised at seeing this given how many eval set workloads we've seen over the past year without evidence of this happening (perhaps it was subtle/small enough that it never went noticed though). If we are actually leaking the full eval log then I'd expect to have seen a much bigger leak in cases where we have > 10GB log files (which we've seen. plenty over as well).



We saw this in our production environment, where a task that was often failing during scoring caused the agent runner pod to be OOM-killed.

`PreviousTask` being the cause is just a guess. I stepped through the code and watched the memory usage, and noticed that it grew at that point and didn't seem to fall back down.

The linked repo should be completely clear of METR stuff. Just a simple agent that dumps a lot of data to the eval log, and a task that randomly fails during scoring.

Okay, we will eventually take a look at this (there are too many other hi-pri things in our backlog to jump on it now). The massive number of eval sets that have been run over the past year w/o this being reported make me think it's not a widespread problem -- if this is a critical blocker for you I recommend that you debug it locally. 

I spent some more time investigating this. I am pretty sure that I misdiagnosed the issue (sorry). There is no memory leak here: the increasing memory usage was due to the retry-functionality succeeding with an increasing number of sample runs.

The issue is rather that this step loads all the sample runs logs into memory at once, which killed our agent runner pods in these runs where we had a large number of samples with large logs.

I will probably propose a PR where we remove the loading of the full logs (rather than just the headers), and just access the full logs when needed to copy them to the new eval run. I don't believe there is any need to have all the full sample run logs loaded into memory at once.