Repository: UKGovernmentBEIS/inspect_ai
Issue: Recording pauses for rate-limiting and transient errors
Root cause: Unimplemented Feature Gap

When measuring how long it takes for a model to complete a task, it's important to factor out rate limits and recovery from other transient errors. We want to be able to both report total actual time taken as well as have time limits enforced based on this actual time.

One significant confounder here is that its not enough to measure when we are backing off, most model APIs do their own internal backoff which is not visible to callers. For example, here is Anthropic's retry handling:

https://github.com/anthropics/anthropic-sdk-python/blob/af3912c5dd97fa821bc6cc8539ee835a28a96edc/src/anthropic/_base_client.py#L1654

This in turn uses a set of custom HTTP headers to determine retry intervals: https://github.com/anthropics/anthropic-sdk-python/blob/af3912c5dd97fa821bc6cc8539ee835a28a96edc/src/anthropic/_base_client.py#L1654

If we want to know about these retries we need to somehow hook into the http client at a much lower level. This will need to be done per-model provider (as e.g Google, Bedrock, and Azure all use bespoke HTTP clients and strategies). 

Related: https://github.com/UKGovernmentBEIS/inspect_ai/issues/1263

There are at least two, possibly separable components:
* Not counting time spent retrying against the agent's run clock
* Recording those pauses in the transcript

Might https://github.com/UKGovernmentBEIS/inspect_ai/pull/1372 make it easier to detect and record pauses in the transcript?

Yes, we now have an exact measure of how much "pausing" occurred for a given model call. What about if we add `waiting` field to `ModelEvent` that records this? Would that be sufficient?

Note that we already report `total_time` and `working_time` for each sample. So we know the "pause time" from that. We don't really have a clean way to exactly pinpoint where pauses happen in model calls as there are different layers or retry and the control flow is not in one top-down stack. We do know at the very end (after the `ModelEvent` is logged) how much waiting time there was, and we record that and use it to compute `working_time`.

So granular pause events are kind of hairy. Hopefully the most important thing is to know how much working time was spent on the whole sample vs. exactly what points in the transcript pauses occurred.

Put another way, we really just have the total time for a series of model requests (as there could be retries that escape from the model libraries) and the time for the successful request. So we can keep the ledger up to date but can't really have a blow-by-blow breakdown of where the pauses occurred.

I'm going to close this as complete based on the `total_time` / `working_time` reported for each sample. Trying to add explicit pause events will be a bit hairy given how our model calls are orchestrated and pause events for semaphores could easily overwhelm the transcript e.g. each and every model call could have a 0.5 second "pause event" and each tool call could also have a 0.5 second "pause event" (both waiting on the resource pools). We could easily end up in a place where the transcript is mostly pause events of very small duration.


I disagree, this issue is not yet addressed. Reviewing the diff of #1372, I don't think we yet have a good way to associate each event in the transcript with the working time at which that event was logged. E.g. what was the `working_time` when each intermediate score, tool, etc. entry was logged. I think this is the real requirement, not necessarily recording explicit pauses.

As a side note, I don't think recording pauses would be that complicated, since:
* we know when tool/model calls are happening
* we can record the total and working time before and after each call
* so we can back out an effective "pause" that starts when the tool was invoked and lasts for a duration equal to the difference between the increases in total and working time

Okay, I get it! So we want each event to include the `time` and `working_time` at which it was logged. One complexity here is that we actually log model and tool events immediately when they start (so that the UI can show that they are in progress) then later amend them when they finish. I am pretty this this can be reconciled with the above but just need to verify on the ground. I'll hopefully have another PR up with these changes later today or worst case tomorrow.

I misspoke before. The `working_time` when the event was initiated (not when it was logged) is what I think is more relevant. E.g. when the tool call was made, not when it finished.

Knowing the duration of events like model generations and tool calls is also pretty valuable, though

Okay, I'll see how granular I can make it. As I said before there is some weird control flow around model calls (partial events for the UI, tenacity handlers, etc.) that make some of this tricky, but I'll try to find a path to recording all of the relevant timing info.


Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1415