Repository: UKGovernmentBEIS/inspect_ai
Issue: eval_set only retries "cancelled" or "errored" logs
Root cause: Validation Gap

Inspect only retries "cancelled" or "errored" logs. This is a problem if you want to retry a run which was not shut down gracefully (most saliently, by the OOM killer in the OS).

We have a work item that we'll be addressing in the short term to make retry handling more customisable. I think we can handle this well in that context. One note is that "started" logs which you have here can also be cruft from runs gone complete bad (e.g. that error on every sample) so re-running them is not a no-brainer (you could end up in an infinite retry loop). 

>We have a work item that we'll be addressing in the short term to make retry handling more customisable. I think we can handle this well in that context. One note is that "started" logs which you have here can also be cruft from runs gone complete bad (e.g. that error on every sample) so re-running them is not a no-brainer (you could end up in an infinite retry loop).

I would have thought that inspect would only mark an EvalLog as "started" if it hasn't been cancelled or seen an error. Surely if every sample errors the log would have status as succeeded?

I think you are correct that the case I identified is not really about samples failing but about the whole eval failing due to a hard crash. Sometimes this would OOM and sometimes it could be something else (which we wouldn't want to keep retrying). I think there is a good resolution to be had here but I don't want to just slam in retrying "started" without thinking carefully about what could happen. There is another discussion about this w/ more details that I'll try to dig up and post a link to here.

>I think you are correct that the case I identified is not really about samples failing but about the whole eval failing due to a hard crash. Sometimes this would OOM and sometimes it could be something else (which we wouldn't want to keep retrying). I think there is a good resolution to be had here but I don't want to just slam in retrying "started" without thinking carefully about what could happen. There is another discussion about this w/ more details that I'll try to dig up and post a link to here.

I'm happy for a setting. The situations I use it in is when I know that inspect has been killed by the OS, so theres an eval log which never finished (is still "started"). I'm then happy to restart it safely in those cases - agreed that perhaps restarting it always could be bad?

Here's the discussion from Slack along w/ a related change we made:

**J.J.**
Looking at the code, I think that could indeed occur in the case of a crash! I will code around that case
https://github.com/UKGovernmentBEIS/inspect_ai/commit/f1c8cbab0fa84c253e84f4f86a52bd113e52a410

**Gabe**
I got this when there were some runs that hard failed such that logs with the "started" state were not properly removed. Removing those logs fixed it, but as a bandaid you could add a check for len(id_logs) and continue if it's empty

**J.J.**
Yup, I just added that band aid to continue w/ len(id_logs) == 0, thx!

**Gabe**
The band-aid may do more harm than good though, since IIUC those "started" log files will prevent eval-set from rerunning those failed runs. It would be nice to have some way of distinguishing started logs that are still going from started logs that are zombies because e.g. a machine crashed. Not sure what would be best, though, maybe some health-check thing and then cleaning "started" logs with last check times older than some threshold?

**J.J.**
Interesting. In theory though that code only runs in between eval calls, so if the task is still running it is from some other process. In other words, at that point in the code they should all be zombies b/c we are outside of eval() which is where they would be running.

**Gabe**
Ah, so you can just clean up all "started" logs if eval-set is called again, since users should not call eval-set while the same eval-set is running? In that case, maybe you could just remove the line a few lines up from this error line that filters out "started" tasks from pruning.

**J.J.**
We are actually expressly trying to ignore the "started" files (we don't want them counted as the most recent eval). We are probably "leaking" them on purpose as a post-flight black box for crashes? (I don't exactly recall, but I don't want to just start changing stuff in there w/o thinking hard about it!)

So it sounds like we are purposely ignoring "started" so that there is some record of the crash to examine post-facto. If we believe that this is unlikely to be useful, then I could see just treating "started" the same as "error" or "cancelled" -- I just recall when creating the feature thinking hard about this and going the other way (I easily could have just not been thinking carefully enough). I'd be game for an immediate change but I'd love it if we could think it through sufficiently so we don't inadvertently break something. Maybe a realtime discussion of this sometime tomorrow just as a sanity check?

It's a good point that we might want to have some evidence of the crash post-facto. I think that in many cases this is a desired behaviour (in particular if there was a run-ending crash). But I think probably needs to be squared w/ the case where it didn't exit gracefully.

Maybe we could keep the header?

Is there any plan to allow (per sample) retries based on a scorer's result? Or does this feature exist and I missed it?




No, there aren't scoring based retries. Two ways you could do this manually:

1. Just run for multiple epochs and use one of the epoch oriented score reducers (https://inspect.ai-safety-institute.org.uk/scorers.html#sec-reducing-epochs)
2. Call the `score()` function from within a solver (looping over however many times to provide multiple retries)


For now, I got it working monkeypatching inspect, would you be interested in a PR with this feature?


If you just post a link to what you did here that will be great (we have a bigger work item to create a retry-policy mechanism that will cover this and other cases, would rather do this work in that context).

The retry behavior should be addressed here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1116. We now make "started" logs eligible for retry (but still screen them out of cleanup so they are still available for post-mortem debugging)