Repository: UKGovernmentBEIS/inspect_ai
Issue: Inspect model API does not record sample waiting time until retries are done, so working time limit can be incorrectly exceeded during prolonged sequences of retries
Root cause: Algorithmic Error

In Inspect's model API, `report_sample_waiting_time()` is called only after the `retry()`ed `generate()` function finally returns (see [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/model/_model.py#L817)). Since Inspect retries an [unlimited number of times](https://inspect.aisi.org.uk/options.html#model-generation:~:text=Maximum%20number%20of%20times%20to%20retry%20generate%20request%20(defaults%20to%20unlimited)) by default, this means that in a sample with a working time limit, if a model never succesfully returns a response due to timeouts or rate-limiting, the model API will never record any waiting time, and so the working time limit will be hit even though it should not be (because all the time was spent waiting for failed model calls).

Minimal example to demonstrate the problem:

```python
import asyncio
import time

from inspect_ai import Task, eval
from inspect_ai.dataset import Sample
from inspect_ai.log import transcript
from inspect_ai.model import (
    ChatMessage,
    GenerateConfig,
    Model,
    ModelAPI,
    ModelOutput,
    modelapi,
)
from inspect_ai.tool import ToolChoice, ToolInfo
from inspect_ai.util import sample_limits


@modelapi(name="fail-n-times")
class FailNTimesAPI(ModelAPI):
    def __init__(self, model_name: str = "fail-n-times", **kwargs):
        super().__init__(model_name)
        self.retries = kwargs["retries"]
        self.retry_count = 0
        self.current_time = time.monotonic()
        self.real_working_time = 0.0
        self.start_time = time.monotonic()

    async def generate(
        self,
        input: list[ChatMessage],
        tools: list[ToolInfo],
        tool_choice: ToolChoice,
        config: GenerateConfig,
    ) -> ModelOutput:
        call_start_time = time.monotonic()
        await asyncio.sleep(0.5)
        self.current_time = time.monotonic()
        self.real_working_time += self.current_time - call_start_time
        self.retry_count += 1
        msg = (
            f"Elapsed time: {self.current_time - self.start_time:.2f} seconds; "
            f"Inspect 'working time' elapsed: {sample_limits().working.usage:.2f} seconds; "
            f"Real working time elapsed: {self.real_working_time:.2f} seconds; "
            f"Retry count: {self.retry_count}/{self.retries}"
        )
        if self.retry_count < self.retries:
            msg += "; now retrying..."
            transcript().info(msg)
            raise Exception("Rate limit exceeded (simulated)")
        else:
            msg += "; success!"
            transcript().info(msg)
            return ModelOutput.from_content(
                model=self.model_name,
                content="Hi",
            )

    def should_retry(self, ex: BaseException) -> bool:
        return True


result = eval(
    tasks=Task(dataset=[Sample(input="Hello", target="Hi")]),
    model=Model(api=FailNTimesAPI(retries=5), config=GenerateConfig()),
    max_samples=1,
    working_limit=15,
)
```

I'm interested in putting in a PR for this if I can come up with a suitable test plan.

@pipmc I think this should address this issue: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2929

Could you give this a review though just as an additional check on my thinking?