Repository: UKGovernmentBEIS/inspect_ai
Issue: `GenerateConfig.timeout` not respected for anthropic
Root cause: Configuration Error

The `GenerateConfig.timeout` doesn't seem to be applied when using anthropic models.  It looks like it gets applied for [batched requests but not non-batched requests](https://github.com/UKGovernmentBEIS/inspect_ai/blob/3388e86205ee6fbe861d3d93b1b84ed2b6cdd973/src/inspect_ai/model/_providers/anthropic.py#L335).

MWE for `0.3.136` (this is outside `eval()` but I can confirm that it also happens for real evals when using `--timeout`)
```python
import asyncio
import time

from inspect_ai.model import ChatMessageUser, GenerateConfig, get_model

if __name__ == "__main__":
    m = get_model("anthropic/claude-sonnet-4-20250514")
    messages = [ChatMessageUser(content="I need to test your timeouts.  Write a very long essay about anything you want, but make sure it is at least 20 lines long.")]

    start_time = time.perf_counter()
    response = asyncio.run(m.generate(messages, config=GenerateConfig(timeout=1, max_retries=1)))
    end_time = time.perf_counter()
    print(f"Elapsed time: {end_time - start_time}; content contains {len(response.choices[0].message.content[0].text)} characters")
```

expected: `TimeoutError`
actual:
```
Elapsed time: 18.488788984250277; content contains 4070 characters
```

Less importantly, I think the fact that timeouts are applied on a provider level means that custom ModelAPI subclasses will also not have the `--timeout` option applied unless they explicitly add support for it; perhaps it could be enforced at a higher level of abstraction, though that might require custom apis to provide a `_generate` instead of overriding `generate`.

Yeah I think we are just using `timeout` to bound our tenacity retries but if there are no retries then the timeout is never applied. I think the fix is that we should forward that `timeout` through to Anthropic. We will make this change and report here when its in, but note that timeouts in model interfaces tend to not be absolute limits but more connection/read timeouts, so it might still not fire for your example code.

At a higher level, you can use a sample-level time limit or even a `time_limit()` context manager to have a "hard" upper bound: https://inspect.aisi.org.uk/errors-and-limits.html#time-limit-1

Thanks; I certainly can make things work with a `time_limit()` or even `asyncio.wait_for`, but I think my use case would be best served if there was a `--timeout` that consistently worked as a response timeout on llm calls.  Maybe it's a somewhat niche case, but in the interest of documenting it here:
- I'm primarily trying to guard against the scenario where a single llm call takes a very long or virtually infinite amount of time (api bugs / pathological llm behavior)
- there might be 10-50 calls for a sample, so for sample-level `--time-limit` I'd have to set it at over an hour to be safe.  But then a stuck api call could take an hour to be detected, and I would have to retry the entire sample instead of just the failed call.
- Both `--time-limit` and `with time_limit()` won't benefit from the Inspect's llm call retry settings, so I need to re-implement retry behavior and expose `max_retries` as a solver arg (for every solver I make) instead of using `--max-retries`.  That's what I'm currently doing (helper fn that just adds timeouts/retries around `model.generate`).

That's helpful, thank you. Thinking this through a bit more I think it might be pretty easy to do what your asking: we can just surround the call to the underlying model provider with `asyncio.wait_for` (computing the correct wait time based on previous requests) and then we don't run into the problem with tenacity not enforcing until requests complete. We will work on this later this week and post back here with progress.

Noted that you said:

> here might be 10-50 calls for a sample, so for sample-level --time-limit I'd have to set it at over an hour to be safe. But then a stuck api call could take an hour to be detected, and I would have to retry the entire sample instead of just the failed call.

Note that if we implement this behavior (enforce the time limit at the api.generate call site as well) then you will still need to retry the entire sample. Wondering based on the above whether you want something different -- a model call timeout that's distinct from the overall `generate()` timeout? (so just results in a normal tenacity retry). If that's the case I think we would need to introduce a new setting here.

LMK the precise behavior you are hoping for and hopefully we can converge from there.


Yes, my preference would be that the timeout just retries the specific call that timed out rather than failing the whole sample.  If it fails the whole sample it would at least let me do sample-level retries with a more aggressive limit than `--time-limit`, but it's still more efficient if I can just retry the one timed-out request (and then use max-retries to control when to fail the entire sample).

Okay, but that would change the semantics of `timeout` to "timeout per request" from "timeout across all retries" (which definitely could break existing workflows that depend on it as a timeout across all retries).

We could add a `request_timeout` which controls only per-request timeouts?

> We could add a `request_timeout` which controls only per-request timeouts?

That sounds great to me; makes sense not to change `--timeout` if it wasn't intended to behave that way

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/2600

Awesome, thank you!