Repository: UKGovernmentBEIS/inspect_ai
Issue: Unknown: None Stream removed (w/ GRPC errors)
Root cause: External Dependency Breakage

Sometimes getting this error when I try with the MBPP dataset, using Gemini-1.5-flash/pro, `sandbox=local` on Mac 13.5. 

Getting logs like `I0000 00:00:1726145168.699596 6919962 fork_posix.cc:77] Other threads are currently calling into gRPC, skipping fork() handlers`, not sure how related. I'm able to finish the benchmarks by re-running/retrying.

Top-of-mind possible causes:
- Gemini (prev. getting HTTP 500 errors for 1.5-pro)
- network errors
- env-specific issues w/ local sandbox
 
CC: @jjallaire-aisi 

```
Task Failed: Stream removed
┌───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────┐
│ /Users/jddantes/Projects/others/Interpretability/inspect_ai/src/inspect_ai/_eval/task/run.py:248 in task_run    │
│                                                                                                                      │
│ /Users/jddantes/Projects/others/Interpretability/inspect_ai/src/inspect_ai/_eval/task/run.py:432 in             │
│ task_run_sample                                                                                                      │
│                                                                                                                      │
│ /Users/jddantes/Projects/others/Interpretability/inspect_ai/src/inspect_ai/_eval/task/error.py:22 in __call__   │
│                                                                                                                      │
│ /Users/jddantes/Projects/others/Interpretability/inspect_ai/src/inspect_ai/_eval/task/run.py:404 in             │
│ task_run_sample                                                                                                      │
│                                                                                                                      │
│ /Users/jddantes/Projects/others/Interpretability/inspect_ai/src/inspect_ai/solver/_solver.py:236 in solve       │
│                                                                                                                      │
│ /Users/jddantes/Projects/others/Interpretability/inspect_ai/src/inspect_ai/_eval/task/run.py:211 in generate    │
│                                                                                                                      │
│ /Users/jddantes/Projects/others/Interpretability/inspect_ai/src/inspect_ai/_eval/task/generate.py:31 in         │
│ task_generate                                                                                                        │
│                                                                                                                      │
│ /Users/jddantes/Projects/others/Interpretability/inspect_ai/src/inspect_ai/model/_model.py:245 in generate      │
│                                                                                                                      │
│ /Users/jddantes/Projects/others/Interpretability/inspect_ai/src/inspect_ai/model/_model.py:367 in _generate     │
│                                                                                                                      │
│ /Users/jddantes/miniconda3/envs/inspect/lib/python3.10/site-packages/tenacity/asyncio/__init__.py:189 in async_wrapped  │
│                                                                                                                      │
│ /Users/jddantes/miniconda3/envs/inspect/lib/python3.10/site-packages/tenacity/asyncio/__init__.py:111 in __call__       │
│                                                                                                                      │
│ /Users/jddantes/miniconda3/envs/inspect/lib/python3.10/site-packages/tenacity/asyncio/__init__.py:153 in iter           │
│                                                                                                                      │
│ /Users/jddantes/miniconda3/envs/inspect/lib/python3.10/site-packages/tenacity/_utils.py:99 in inner                     │
│                                                                                                                      │
│ /Users/jddantes/miniconda3/envs/inspect/lib/python3.10/site-packages/tenacity/__init__.py:398 in <lambda>               │
│                                                                                                                      │
│ /Users/jddantes/miniconda3/envs/inspect/lib/python3.10/concurrent/futures/_base.py:451 in result                        │
│                                                                                                                      │
│   448 │   │   │   │   if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:                                         │
│   449 │   │   │   │   │   raise CancelledError()                                                                     │
│   450 │   │   │   │   elif self._state == FINISHED:                                                                  │
│ > 451 │   │   │   │   │   return self.__get_result()                                                                 │
│   452 │   │   │   │                                                                                                  │
│   453 │   │   │   │   self._condition.wait(timeout)                                                                  │
│   454                                                                                                                │
│                                                                                                                      │
│ /Users/jddantes/miniconda3/envs/inspect/lib/python3.10/concurrent/futures/_base.py:403 in __get_result                  │
│                                                                                                                      │
│   400 │   def __get_result(self):                                                                                    │
│   401 │   │   if self._exception:                                                                                    │
│   402 │   │   │   try:                                                                                               │
│ > 403 │   │   │   │   raise self._exception                                                                          │
│   404 │   │   │   finally:                                                                                           │
│   405 │   │   │   │   # Break a reference cycle with the exception in self._exception                                │
│   406 │   │   │   │   self = None                                                                                    │
│                                                                                                                      │
│ /Users/jddantes/miniconda3/envs/inspect/lib/python3.10/site-packages/tenacity/asyncio/__init__.py:114 in __call__       │
│                                                                                                                      │
│ /Users/jddantes/Projects/others/Interpretability/ASET/inspect_ai/src/inspect_ai/model/_model.py:331 in generate      │
│                                                                                                                      │
│ /Users/jddantes/Projects/others/Interpretability/ASET/inspect_ai/src/inspect_ai/model/_providers/google.py:137 in    │
│ generate                                                                                                             │
│                                                                                                                      │
│ /Users/jddantes/miniconda3/envs/inspect/lib/python3.10/site-packages/google/generativeai/generative_models.py:385 in    │
│ generate_content_async                                                                                               │
│                                                                                                                      │
│   382 │   │   │   │   │   )                                                                                          │
│   383 │   │   │   │   return await generation_types.AsyncGenerateContentResponse.from_aiterato                       │
│   384 │   │   │   else:                                                                                              │
│ > 385 │   │   │   │   response = await self._async_client.generate_content(                                          │
│   386 │   │   │   │   │   request,                                                                                   │
│   387 │   │   │   │   │   **request_options,                                                                         │
│   388 │   │   │   │   )                                                                                              │
│                                                                                                                      │
│ /Users/jddantes/miniconda3/envs/inspect/lib/python3.10/site-packages/google/ai/generativelanguage_v1beta/services/gener │
│ ative_service/async_client.py:406 in generate_content                                                                │
│                                                                                                                      │
│    403 │   │   self._client._validate_universe_domain()                                                              │
│    404 │   │                                                                                                         │
│    405 │   │   # Send the request.                                                                                   │
│ >  406 │   │   response = await rpc(                                                                                 │
│    407 │   │   │   request,                                                                                          │
│    408 │   │   │   retry=retry,                                                                                      │
│    409 │   │   │   timeout=timeout,                                                                                  │
│                                                                                                                      │
│ /Users/jddantes/miniconda3/envs/inspect/lib/python3.10/site-packages/google/api_core/retry/retry_unary_async.py:230 in  │
│ retry_wrapped_func                                                                                                   │
│                                                                                                                      │
│   227 │   │   │   sleep_generator = exponential_sleep_generator(                                                     │
│   228 │   │   │   │   self._initial, self._maximum, multiplier=self._multiplier                                      │
│   229 │   │   │   )                                                                                                  │
│ > 230 │   │   │   return await retry_target(                                                                         │
│   231 │   │   │   │   functools.partial(func, *args, **kwargs),                                                      │
│   232 │   │   │   │   predicate=self._predicate,                                                                     │
│   233 │   │   │   │   sleep_generator=sleep_generator,                                                               │
│                                                                                                                      │
│ /Users/jddantes/miniconda3/envs/inspect/lib/python3.10/site-packages/google/api_core/retry/retry_unary_async.py:160 in  │
│ retry_target                                                                                                         │
│                                                                                                                      │
│   157 │   │   # This function explicitly must deal with broad exceptions.                                            │
│   158 │   │   except Exception as exc:                                                                               │
│   159 │   │   │   # defer to shared logic for handling errors                                                        │
│ > 160 │   │   │   _retry_error_helper(                                                                               │
│   161 │   │   │   │   exc,                                                                                           │
│   162 │   │   │   │   deadline,                                                                                      │
│   163 │   │   │   │   sleep,                                                                                         │
│                                                                                                                      │
│ /Users/jddantes/miniconda3/envs/inspect/lib/python3.10/site-packages/google/api_core/retry/retry_base.py:212 in         │
│ _retry_error_helper                                                                                                  │
│                                                                                                                      │
│   209 │   │   │   RetryFailureReason.NON_RETRYABLE_ERROR,                                                            │
│   210 │   │   │   original_timeout,                                                                                  │
│   211 │   │   )                                                                                                      │
│ > 212 │   │   raise final_exc from source_exc                                                                        │
│   213 │   if on_error_fn is not None:                                                                                │
│   214 │   │   on_error_fn(exc)                                                                                       │
│   215 │   if deadline is not None and time.monotonic() + next_sleep > deadline:                                      │
│                                                                                                                      │
│ /Users/jddantes/miniconda3/envs/inspect/lib/python3.10/site-packages/google/api_core/retry/retry_unary_async.py:155 in  │
│ retry_target                                                                                                         │
│                                                                                                                      │
│   152 │                                                                                                              │
│   153 │   for sleep in sleep_generator:                                                                              │
│   154 │   │   try:                                                                                                   │
│ > 155 │   │   │   return await target()                                                                              │
│   156 │   │   # pylint: disable=broad-except                                                                         │
│   157 │   │   # This function explicitly must deal with broad exceptions.                                            │
│   158 │   │   except Exception as exc:                                                                               │
│                                                                                                                      │
│ /Users/jddantes/miniconda3/envs/inspect/lib/python3.10/site-packages/google/api_core/grpc_helpers_async.py:88 in        │
│ __await__                                                                                                            │
│                                                                                                                      │
│    85 │   │   │   response = yield from self._call.__await__()                                                       │
│    86 │   │   │   return response                                                                                    │
│    87 │   │   except grpc.RpcError as rpc_error:                                                                     │
│ >  88 │   │   │   raise exceptions.from_grpc_error(rpc_error) from rpc_error                                         │
│    89                                                                                                                │
│    90                                                                                                                │
│    91 class _WrappedStreamResponseMixin(Generic[P], _WrappedCall):                                                   │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Unknown: None Stream removed
```

Not sure if you discovered anything further here, please feel free to re-open this with hopefully a cleaner repro/diagnostic and we can see if there something we can do to be more resiliant.

Unfortunately I wasn't able to get back yet to this issue--will just re-open as you mentioned when I'm able to dig deeper and report more details