Repository: UKGovernmentBEIS/inspect_ai
Issue: Non-deterministic error in `proto` package when using Google models
Root cause: External Dependency Breakage

This appears to happen non-deterministically, but often enough to be a big problem (have encountered it ~5 times).

Likely related to Google's protocol buffers.

# Traceback
```
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/__w/benchmarks/benchmarks/bench/cli/run_eval.py", line 62, in <module>
    click_cli()
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/benchmarks/benchmarks/bench/cli/run_eval.py", line 58, in click_cli
    return cli(task_path, provider_path, model_name, airtable_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/benchmarks/benchmarks/bench/cli/run_eval.py", line 45, in cli
    log = eval(task, provider_cls, model_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/benchmarks/benchmarks/bench/__init__.py", line [90](https://github.com/epoch-research/benchmarks/actions/runs/13284823215/job/37091147615#step:7:91), in eval
    logs = inspect_ai.eval(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/eval.py", line 165, in eval
    return task_display().run_task_app(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_display/plain/display.py", line 45, in run_task_app
    return asyncio.run(main)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/eval.py", line 416, in eval_async
    await eval_run(
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/run.py", line 214, in eval_run
    return await run_single(task_run_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/run.py", line 237, in run_single
    return await asyncio.gather(*asyncio_tasks)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 324, in task_run
    sample_results = await asyncio.gather(*sample_coroutines)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 733, in task_run_sample
    error = handle_error(ex)
            ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 542, in handle_error
    err = sample_error(ex)
          ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/error.py", line 22, in __call__
    raise ex
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 653, in task_run_sample
    error = handle_error(ex)
            ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 542, in handle_error
    err = sample_error(ex)
          ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/error.py", line 22, in __call__
    raise ex
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 597, in task_run_sample
    state = await plan(state, generate)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/solver/_plan.py", line 105, in __call__
    state = await solver(state, generate)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/solver/_solver.py", line 278, in solve
    return await generate(state, tool_calls=tool_calls, cache=cache, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 243, in generate
    return await task_generate(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/generate.py", line 31, in task_generate
    state.output = await model.generate(
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_model.py", line 261, in generate
    return await self._generate(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_model.py", line 441, in _generate
    model_output = await generate()
                   ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_model.py", line 388, in generate
    result = await self.api.generate(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_providers/google.py", line 190, in generate
    choices=completion_choices_from_candidates(response.candidates),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_providers/google.py", line 560, in completion_choices_from_candidates
    return [completion_choice_from_candidate(candidate) for candidate in candidates]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_providers/google.py", line 560, in <listcomp>
    return [completion_choice_from_candidate(candidate) for candidate in candidates]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen _collections_abc>", line 9[93](https://github.com/epoch-research/benchmarks/actions/runs/13284823215/job/37091147615#step:7:94), in __iter__
  File "/usr/local/lib/python3.11/site-packages/proto/marshal/collections/repeated.py", line 125, in __getitem__
    return self._marshal.to_python(self._pb_type, self.pb[key])
                                   ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/proto/utils.py", line 50, in inner
    self._cached_values[fx.__name__] = fx(self)
                                       ^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/proto/marshal/collections/repeated.py", line 116, in _pb_type
    canary = copy.deepcopy(self.pb).add()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'add'
```

# Environment
## Full lockfile
Uploaded as .txt because GitHub won't allow arbitrary file types

[poetry.lock.txt](https://github.com/user-attachments/files/18768492/poetry.lock.txt)

## Selected dependencies
```shell
❯ poetry show protobuf
 name         : protobuf 
 version      : 4.25.5   
 description  :          

required by
 - google-ai-generativelanguage >=3.20.2,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0dev
 - google-api-core >=3.19.5,<3.20.0 || >3.20.0,<3.20.1 || >3.20.1,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0.dev0
 - google-cloud-aiplatform >=3.20.2,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0dev
 - google-cloud-resource-manager >=3.20.2,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0dev
 - google-generativeai *
 - googleapis-common-protos >=3.20.2,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0.dev0
 - grpc-google-iam-v1 >=3.20.2,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0dev
 - grpcio-status >=4.21.6
 - mlflow-skinny >=3.12.0,<6
 - proto-plus >=3.19.0,<6.0.0dev
 - sagemaker >=3.12,<5.0
```

```shell
❯ poetry show google-generativeai
 name         : google-generativeai                                           
 version      : 0.8.3                                                         
 description  : Google Generative AI High level API client library and tools. 

dependencies
 - google-ai-generativelanguage 0.6.10
 - google-api-core *
 - google-api-python-client *
 - google-auth >=2.15.0
 - protobuf *
 - pydantic *
 - tqdm *
 - typing-extensions *
```

The line `protobuf *` in dependencies of `google-generativeai` certainly smells like the kind of thing that could be to blame.

We've noticed that in some cases Google has started returning 0 completion choices when there is a refusal. It may be that their type declarations are out of sync with their service (or perhaps they are a/b testing a new service endpoint and that's where the declarations are out of sync). 

Further, Google maddeningly doesn't produce type declarations for their genai packages --- I noticed that under the hood the `candidates` type is `MutableSequence[Candidate]` rather than `list[Candidate]` (which is what our helper function was taking). Unlikely this is of consequence but worth running down every angle.

Finally, Google is in the course of introducing a brand new Python client for Gemini. They have said it is "not ready for production". We nevertheless have an implementation of it internally that we are hoping to promote soon (but we've observed numerous issues with it).

Here is a speculative fix for the candidates issues: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1302

>  I noticed that under the hood the candidates type is MutableSequence[Candidate] rather than list[Candidate] (which is what our helper function was taking). Unlikely this is of consequence but worth running down every angle.

Mmh, this would make sense as the source of the error if it was `MutableSet` (because the error is calling `.add()` on a list). But `MutableSequence` should be fine, I think. 

https://docs.python.org/3/library/collections.abc.html#collections-abstract-base-classes

Yeah, I saw that it was fine. The `MutableSequence` type doesn't have a `.sort()` method (which we were calling without an exception) so my best guess is that it was a list anyway. In any case, I did update the types and move to the `sorted()` method. The speculative fix is really to check for a falsey value on `candidates` (which we'd get for a null value or empty list).



With #1302 (i.e. I am running this on ec5c6f073c28), I now get an error that suggests the problem is due to 0 completion choices returned. Still can't share a repro due to non-determinism. 

```
2025-02-17T22:21:57.2378149Z Traceback (most recent call last):
2025-02-17T22:21:57.2386187Z   File "<frozen runpy>", line 198, in _run_module_as_main
2025-02-17T22:21:57.2386992Z   File "<frozen runpy>", line 88, in _run_code
2025-02-17T22:21:57.2387845Z   File "/__w/benchmarks/benchmarks/bench/cli/run_eval.py", line 62, in <module>
2025-02-17T22:21:57.2388612Z     click_cli()
2025-02-17T22:21:57.2389399Z   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
2025-02-17T22:21:57.2390210Z     return self.main(*args, **kwargs)
2025-02-17T22:21:57.2390735Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2391485Z   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
2025-02-17T22:21:57.2392264Z     rv = self.invoke(ctx)
2025-02-17T22:21:57.2392735Z          ^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2393639Z   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
2025-02-17T22:21:57.2394502Z     return ctx.invoke(self.callback, **ctx.params)
2025-02-17T22:21:57.2395105Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2409572Z   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
2025-02-17T22:21:57.2418096Z     return __callback(*args, **kwargs)
2025-02-17T22:21:57.2426990Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2428054Z   File "/__w/benchmarks/benchmarks/bench/cli/run_eval.py", line 58, in click_cli
2025-02-17T22:21:57.2429108Z     return cli(task_path, provider_path, model_name, airtable_id)
2025-02-17T22:21:57.2431826Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2433561Z   File "/__w/benchmarks/benchmarks/bench/cli/run_eval.py", line 45, in cli
2025-02-17T22:21:57.2434624Z     log = eval(task, provider_cls, model_name)
2025-02-17T22:21:57.2435343Z           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2436269Z   File "/__w/benchmarks/benchmarks/bench/__init__.py", line 124, in eval
2025-02-17T22:21:57.2437101Z     logs = inspect_ai.eval(
2025-02-17T22:21:57.2437709Z            ^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2438636Z   File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/eval.py", line 165, in eval
2025-02-17T22:21:57.2439881Z     return task_display().run_task_app(
2025-02-17T22:21:57.2440902Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2441999Z   File "/usr/local/lib/python3.11/site-packages/inspect_ai/_display/plain/display.py", line 45, in run_task_app
2025-02-17T22:21:57.2443280Z     return asyncio.run(main)
2025-02-17T22:21:57.2448244Z            ^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2448912Z   File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
2025-02-17T22:21:57.2449621Z     return runner.run(main)
2025-02-17T22:21:57.2450086Z            ^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2450689Z   File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
2025-02-17T22:21:57.2451419Z     return self._loop.run_until_complete(task)
2025-02-17T22:21:57.2451972Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2452737Z   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
2025-02-17T22:21:57.2453723Z     return future.result()
2025-02-17T22:21:57.2454173Z            ^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2454973Z   File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/eval.py", line 424, in eval_async
2025-02-17T22:21:57.2455815Z     await eval_run(
2025-02-17T22:21:57.2456535Z   File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/run.py", line 214, in eval_run
2025-02-17T22:21:57.2457459Z     return await run_single(task_run_options)
2025-02-17T22:21:57.2458582Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2459518Z   File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/run.py", line 237, in run_single
2025-02-17T22:21:57.2460387Z     return await asyncio.gather(*asyncio_tasks)
2025-02-17T22:21:57.2460927Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2461748Z   File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 328, in task_run
2025-02-17T22:21:57.2462697Z     sample_results = await asyncio.gather(*sample_coroutines)
2025-02-17T22:21:57.2469966Z                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2470894Z   File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 783, in task_run_sample
2025-02-17T22:21:57.2471776Z     raise raise_error
2025-02-17T22:21:57.2472583Z   File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 609, in task_run_sample
2025-02-17T22:21:57.2473671Z     state = await plan(state, generate)
2025-02-17T22:21:57.2474176Z             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2475003Z   File "/usr/local/lib/python3.11/site-packages/inspect_ai/solver/_plan.py", line 105, in __call__
2025-02-17T22:21:57.2475854Z     state = await solver(state, generate)
2025-02-17T22:21:57.2476358Z             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2477164Z   File "/usr/local/lib/python3.11/site-packages/inspect_ai/solver/_solver.py", line 279, in solve
2025-02-17T22:21:57.2478206Z     return await generate(state, tool_calls=tool_calls, cache=cache, **kwargs)
2025-02-17T22:21:57.2478942Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2479834Z   File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 247, in generate
2025-02-17T22:21:57.2480677Z     return await task_generate(
2025-02-17T22:21:57.2481183Z            ^^^^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2482060Z   File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/generate.py", line 42, in task_generate
2025-02-17T22:21:57.2483188Z     message = state.output.message
2025-02-17T22:21:57.2483696Z               ^^^^^^^^^^^^^^^^^^^^
2025-02-17T22:21:57.2484522Z   File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_model_output.py", line 131, in message
2025-02-17T22:21:57.2485399Z     return self.choices[0].message
2025-02-17T22:21:57.2485869Z            ~~~~~~~~~~~~^^^
2025-02-17T22:21:57.2486334Z IndexError: list index out of range
```

Okay, I think our best recourse is to synthesize a response (rather than plumb through a new assumption that "completions may not have a completion" which is likely just an error in their service that will presumably be resolved soon). This change does that: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1344 (effectively turning it into a refusal with stop reason "unknown").

We are switching to the new google-genai client later this week so hopefully these issues will not exist there (they are now recommending that everyone use the new client in spite of it being a bit raw -- I think for the next few weeks things will be a bit rocky in google land until they get everything buttoned up with the new models and new client).



I think I disprefer https://github.com/UKGovernmentBEIS/inspect_ai/pull/1344 actually. I'd rather fail loudly than introduce such a hack. Sorry if I have the opposite impression above. 

I think I'll just wait until you make the switch to the new client, hopefully that goes well, and then I can update the revision of Inspect I'm using.

It's not a hack. They actually document that they can return no completions (in the case of a refusal). The new client provides additional data on the refusal which we will use, but it is not at all incorrect to return "unable to generate a response" when that documented condition occurs. 

Okay, perhaps I misunderstood. What seemed hacky to me is to return a 

```python
ChatCompletionChoice(
    message=ChatMessageAssistant(
        content="I was unable to generate a response.",
        source="generate",
    ),
    stop_reason="unknown",
)
```

Can you not represent "refusal" in Inspect somehow, without needing the additional data on the refusal? 

Does the object returned from Google not tell you that it's a refusal in any way? If it doesn't tell you anything about a refusal, but just doesn't return a completion, I would probably prefer to fail loudly. But no need to do that just for me.

Yes you can say `stop_reason="content_filter"`. The old API had no info about why there are no choices (but did it anyway sometimes). I thought it better to classify it as "unknown" since it's kind of speculation that its a content filter. With the new API  they have added a field indicating that it is a refusal along with the reason. Even with the new API though its debatable whether to fail loudly or not, as if there are other reasons they might not return completions -- if we raise we make it impossible to complete an eval on that dataset without an error. As they've already demonstrated that they are willing to have "no completions" occur w/ no explanation, there is no guarantee they won't do this in the future. It would be a better world if all of the model APIs were exhaustively specified and fully deterministic but they simply aren't, so we need to make some provision to navigate around the vagaries, inconsistencies, and bugs in these services.

> The old API had no info about why there are no choices (but did it anyway sometimes).

I see, I didn't realise this. 

> Even with the new API though its debatable whether to fail loudly or not

No, if it tells you it's a refusal I very much don't wait to fail loudly! In that case I'd want to represent refusal using an appropriate object in Inspect. 


> No, if it tells you it's a refusal I very much don't wait to fail loudly! In that case I'd want to represent refusal using an appropriate object in Inspect.

I meant in the else clause after we've determined there are no completions AND there is no refusal. That's an ambiguous place which could be another undocumented "no completions" behaviour or could be a service error. Hard to say and there isn't really any way to find out short of having access to the gemini serving source code.



Got it. Yes, in that case my view would be to fail loudly. 

I want to discuss with you how to represent refusals, but that's a bit off-topic now for this issue, so I opened: https://github.com/UKGovernmentBEIS/inspect_ai/issues/1345

Should be resolved with this: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1349