Repository: UKGovernmentBEIS/inspect_ai
Issue: Assertion error after OpenAI refusal
Root cause: Validation Gap

I get an error `AssertionError: OpenAI ChatMessageAssistant internal must be an _AssistantInternal` on a substantial number (>50%) of long trajectory agentic tasks with o3-mini, around 100k-2.5M total tokens.

I believe the issue is caused by OpenAI flagging the prompt as "Invalid" and returning an unexpected response that Inspect fails to handle gracefully.

It would slightly improve the user experience in Inspect if this error was handled slightly more gracefully and the sample was marked as a failure for some specific reason as opposed to triggering an assertion.

It would significantly improve my evals if inspect had a way to roll back to earlier in the conversation when this happens and see if the error can be avoided. I think I could probably build a custom Solver to look for this exact message, rewrite some chat history, and continue, but am not sure if a one-off fix in my code would be best or something I could contribute back to inspect.

```
        {
          "id": "2KDSaoyv4mXwidrQ4n5JGE",
          "content": "",
          "role": "tool",
          "tool_call_id": "call_XDscZwazA5TGTK4JpjglxSw5",
          "function": "bash"
        },
        {
          "id": "GrYAhEsqrkcvkTMk6aHqVx",
          "content": "Invalid prompt: your prompt was flagged as potentially violating our usage policy. Please try again with a different prompt: https://platform.openai.com/docs/guides/reasoning#advice-on-prompting",
          "source": "generate",
          "role": "assistant",
          "model": "o3-mini-2025-01-31"
        },
        {
          "id": "DWR4wTFLoGCrZToyJVTgnx",
          "content": "\nPlease proceed to the next step using your best judgement. If you believe you\nhave completed the task, please call the `submit()` tool.\n",
          "role": "user"
        }
```

Traceback:
```
┌───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────┐
│ /home/dev/git/inspect_ai/src/inspect_ai/_eval/task/run.py:667 in task_run_sample                                     │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/solver/_plan.py:106 in __call__                                              │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/agent/_as_solver.py:56 in solve                                              │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/agent/_react.py:167 in execute                                               │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/agent/_react.py:309 in _agent_generate                                       │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/agent/_react.py:314 in generate                                              │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/model/_model.py:400 in generate                                              │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/model/_model.py:685 in _generate                                             │
│                                                                                                                      │
│ /home/dev/git/my_repo_removed/.venv/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:189 in async_wrapped  │
│                                                                                                                      │
│ /home/dev/git/my_repo_removed/.venv/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:111 in __call__       │
│                                                                                                                      │
│ /home/dev/git/my_repo_removed/.venv/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:153 in iter           │
│                                                                                                                      │
│ /home/dev/git/my_repo_removed/.venv/lib/python3.12/site-packages/tenacity/_utils.py:99 in inner                     │
│                                                                                                                      │
│ /home/dev/git/my_repo_removed/.venv/lib/python3.12/site-packages/tenacity/__init__.py:400 in <lambda>               │
│                                                                                                                      │
│ /home/dev/.local/share/uv/python/cpython-3.12.7-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py:449 in   │
│ result                                                                                                               │
│                                                                                                                      │
│   446 │   │   │   │   if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:                                         │
│   447 │   │   │   │   │   raise CancelledError()                                                                     │
│   448 │   │   │   │   elif self._state == FINISHED:                                                                  │
│ > 449 │   │   │   │   │   return self.__get_result()                                                                 │
│   450 │   │   │   │                                                                                                  │
│   451 │   │   │   │   self._condition.wait(timeout)                                                                  │
│   452                                                                                                                │
│                                                                                                                      │
│ /home/dev/.local/share/uv/python/cpython-3.12.7-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py:401 in   │
│ __get_result                                                                                                         │
│                                                                                                                      │
│   398 │   def __get_result(self):                                                                                    │
│   399 │   │   if self._exception:                                                                                    │
│   400 │   │   │   try:                                                                                               │
│ > 401 │   │   │   │   raise self._exception                                                                          │
│   402 │   │   │   finally:                                                                                           │
│   403 │   │   │   │   # Break a reference cycle with the exception in self._exception                                │
│   404 │   │   │   │   self = None                                                                                    │
│                                                                                                                      │
│ /home/dev/git/my_repo_removed/.venv/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:114 in __call__       │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/model/_model.py:625 in generate                                              │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/model/_providers/openai.py:245 in generate                                   │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/model/_providers/openai_responses.py:64 in generate_responses                │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/model/_openai_responses.py:64 in openai_responses_inputs                     │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/model/_openai_responses.py:87 in _openai_input_item_from_chat_message        │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/model/_openai_responses.py:312 in                                            │
│ _openai_input_items_from_chat_message_assistant                                                                      │
│                                                                                                                      │
│ /home/dev/git/inspect_ai/src/inspect_ai/model/_openai_responses.py:424 in _ids_from_assistant_internal               │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
AssertionError: OpenAI ChatMessageAssistant internal must be an _AssistantInternal
```

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1755