Repository: UKGovernmentBEIS/inspect_ai
Issue: OpenRouter returns responses with no choices, causing TypeError
Root cause: Validation Gap

Running evals via OpenRouter with some models/providers throws a `TypeError` on a "Too Many Requests" response.

Reproduced with: `inspect eval inspect_evals/mmmu --model=openrouter/qwen/qwen-2-vl-72b-instruct`
(But I'd recommend a different eval since mmmu takes ages to download the dataset and doesn't cache it)

Dump of `response.json()` from `chat_choices_from_openai(response)`:
```json
{
    "id": null,
    "choices": null,
    "created": null,
    "model": null,
    "object": null,
    "service_tier": null,
    "system_fingerprint": null,
    "usage": null,
    "error": {
        "message": "Provider returned error",
        "code": 429,
        "metadata": {
            "raw": "Too Many Requests",
            "provider_name": "Hyperbolic"
        }
    },
    "user_id": "user_xyz"
}
```

Traceback:
```
┌───────────────────── Traceback (most recent call last) ─────────────────────┐
│ /opt/venv/lib/python3.12/site-packages/inspect_ai/_eval/task/run.py:626 in  │
│ task_run_sample                                                             │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/inspect_ai/solver/_plan.py:106 in    │
│ __call__                                                                    │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/inspect_ai/solver/_multiple_choice.p │
│ y:302 in solve                                                              │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/inspect_ai/_eval/task/run.py:249 in  │
│ generate                                                                    │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/inspect_ai/_eval/task/generate.py:33 │
│ in task_generate                                                            │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/inspect_ai/model/_model.py:346 in    │
│ generate                                                                    │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/inspect_ai/model/_model.py:561 in    │
│ _generate                                                                   │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:189 in  │
│ async_wrapped                                                               │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:111 in  │
│ __call__                                                                    │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:153 in  │
│ iter                                                                        │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/tenacity/_utils.py:99 in inner       │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/tenacity/__init__.py:398 in <lambda> │
│                                                                             │
│ /usr/lib/python3.12/concurrent/futures/_base.py:449 in result               │
│                                                                             │
│   446 │   │   │   │   if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED] │
│   447 │   │   │   │   │   raise CancelledError()                            │
│   448 │   │   │   │   elif self._state == FINISHED:                         │
│ > 449 │   │   │   │   │   return self.__get_result()                        │
│   450 │   │   │   │                                                         │
│   451 │   │   │   │   self._condition.wait(timeout)                         │
│   452                                                                       │
│                                                                             │
│ /usr/lib/python3.12/concurrent/futures/_base.py:401 in __get_result         │
│                                                                             │
│   398 │   def __get_result(self):                                           │
│   399 │   │   if self._exception:                                           │
│   400 │   │   │   try:                                                      │
│ > 401 │   │   │   │   raise self._exception                                 │
│   402 │   │   │   finally:                                                  │
│   403 │   │   │   │   # Break a reference cycle with the exception in self. │
│   404 │   │   │   │   self = None                                           │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:114 in  │
│ __call__                                                                    │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/inspect_ai/model/_model.py:501 in    │
│ generate                                                                    │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/inspect_ai/model/_providers/openai.p │
│ y:224 in generate                                                           │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/inspect_ai/model/_providers/openai.p │
│ y:257 in _chat_choices_from_response                                        │
│                                                                             │
│ /opt/venv/lib/python3.12/site-packages/inspect_ai/model/_openai.py:468 in   │
│ chat_choices_from_openai                                                    │
└─────────────────────────────────────────────────────────────────────────────┘
TypeError: 'NoneType' object is not iterable
```

Code snippet from `src/inspect_ai/model/_openai.py`:
```python
def chat_choices_from_openai(
    response: ChatCompletion, tools: list[ToolInfo]
) -> list[ChatCompletionChoice]:
    choices = list(response.choices)  #  <====================================== Boom
    choices.sort(key=lambda c: c.index)
    return [
        ChatCompletionChoice(
            message=chat_message_assistant_from_openai(choice.message, tools),
            stop_reason=as_stop_reason(choice.finish_reason),
            logprobs=(
                Logprobs(**choice.logprobs.model_dump())
                if choice.logprobs is not None
                else None
            ),
        )
        for choice in choices
    ]
```

Okay, here I've attempted to handle these errors in the ordinary flow so they get the requisite retry treatment: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1536/files

Note that the docs at https://openrouter.ai/docs/api-reference/errors indicate that returning "no response" is also possible even w/o an error if they model/provider is being warmed up so that is handled as well.

Could you give this a try w/ your scenario to ensure that it works as expected? You can install from `main` with:

``` bash
pip install git+https://github.com/UKGovernmentBEIS/inspect_ai
```




Yep that seems to fix it alright!
Thanks for the quick work!
There's a [related issue for openrouter](https://github.com/UKGovernmentBEIS/inspect_evals/issues/281) I raised on the inspect_evals repo but I think it'll need addressing in inspect_ai.