Repository: UKGovernmentBEIS/inspect_ai
Issue: Agentic evals on Mistral Large hosted on Azure fail
Root cause: Interface Contract Mismatch

I'm hosting Mistral Large on Azure in a serverless deployment. On agentic evals (e.g. `inspect eval inspect_evals/cybench -T challenges=data_siege`), I occasionally encounter the following error:

```
RuntimeError: HttpResponseError('(Bad Request) {"object":"Error","message":"Unexpected role \'user\' after role \'tool\'","type":"invalid_request_error","code":3230}\nCode: Bad Request\nMessage: {"object":"Error","message":"Unexpected role \'user\' after role \'tool\'","type":"invalid_request_error","code":3230}') 
```

This happens (I think) when the agent submits an incorrect answer since in those cases the sequence of calls is:

* submit tool
* user message saying "Your submission was incorrect. Please proceed and attempt to find the correct answer."

I've attached the json log of a failed run. 


[eval.json](https://github.com/user-attachments/files/20055288/eval.json)

I'll be taking a look at Mistral Large today and tomorrow, will let you know if I figure this out or can work around it in the provider or whether it needs to be worked around in the agents.

I was able to repro this when the agent submits an incorrect answer because of the injected user message. I think your hypothesis is correct. For example, this little eval test case repros it pretty consistently:
```python
from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import includes
from inspect_ai.solver import basic_agent
from inspect_ai.tool import Tool, tool


@tool
def get_answer() -> Tool:
    async def execute():
        """Gets the answer."""
        return "tool-used"

    return execute


@task
def basic_tool_task() -> Task:
    return Task(
        dataset=[
            Sample(
                input="Use the get_answer tool and submit the WRONG answer once before submitting the RIGHT answer. Note that the submit answer tool has a similar format to the get_answer tool, so make sure to pay attention to the formatting.",
                target="tool-used",
            )
        ],
        solver=basic_agent(
            tools=[get_answer()],
            max_attempts=3,
            message_limit=30,
        ),
        scorer=includes(),
    )
```

Some models, including Mistral, expect message roles in a certain order. In in this case, `basic_agent` (and the new ReAct agents) will append a user message after an incorrect tool call if allowed multiple attempts. `tool` messages followed by `user` messages are not what Mistral Large expects.

There are a few ways of addressing this:
1. The easiest way to deal with this is to just not allow multiple attempts. However, this requires modification of eval semantics which is generally not desirable.
2. The Mistral provider [handles this by folding user messages into the tool messages](https://github.com/UKGovernmentBEIS/inspect_ai/blob/a5b809afc0a339116b4a58910329099a89c2b1ac/src/inspect_ai/model/_providers/mistral.py#L320-L359) when they appear in the invalid order. 

I suspect the cleanest way to solve this is to try and back-port the Mistral chat folding behavior into the Azure provider. 

Unfortunately, I haven't found a non-hacky way of using the MistralAI provider for Azure-hosted models (I think there was a recent regression with Azure, it requires patching some of the code used to construct the full URLs.)