Repository: UKGovernmentBEIS/inspect_ai
Issue: AzureAI might be broken
Root cause: Configuration Error

A colleague and I have been running Inspect Evals against models deployed using the [AzureAI provider](https://inspect.ai-safety-institute.org.uk/providers.html#azure-ai). About a month ago, this stopped working and we were getting the following error:

```
│ ResourceNotFoundError: (404) Resource not found                                                                                                                                 │
│ Code: 404   
```

We redeployed which did not fix things. However, the following Python script DOES work, so we know our deployment exists and can be queried, but we've been unable to get it working via Inspect :(

```
import os
from openai import AzureOpenAI
endpoint = "https://ft-test-sweden.openai.azure.com/"
model_name = "o3-mini"
deployment = "o3-mini"
subscription_key = "<snip>"
api_version = "2024-12-01-preview"
client = AzureOpenAI(
    api_version=api_version,
    azure_endpoint=endpoint,
    api_key=subscription_key,
)
response = client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant.",
        },
        {
            "role": "user",
            "content": "I am going to Paris, what should I see?",
        }
    ],
    max_completion_tokens=100000,
    model=deployment
)
print(response.choices[0].message.content)
```

I tried downgrading Inspect to 0.3.45 and got the same error, so I guess the problem is with the Azure SDK or similar?

The command I'm running is `inspect eval-set eval.py  --log-dir eval-set-logs --model=azureai/o3-mini`, with `AZUREAI_API_KEY` and `AZUREAI_BASE_URL` in my `.env`. I've also tried various other environment variables, and also hard-coding the relevant parts of `models/_providers/azureai.py`, but without ripping out the guts of the ChatCompletionsClient this hasn't got us unblocked.

I've been using `0.3.62`, my colleague `0.3.74`, so kind of doubtful it's been broken in an Inspect update

One comment is that the preferred method for accessing OpenAI on Azure is actually to use the OpenAI interface w/ the `azure/` qualifier: https://inspect.ai-safety-institute.org.uk/providers.html#openai-on-azure (which would match with the way you are accessing it above).




> One comment is that the preferred method for accessing OpenAI on Azure is actually to use the OpenAI interface w/ the `azure/` qualifier: https://inspect.ai-safety-institute.org.uk/providers.html#openai-on-azure (which would match with the way you are accessing it above).

Interesting. I did try that yesterday as well, based on someone's suggestion in the Slack :( Just tried again as well; the only variables I'm passing in are this `.env`, as well

```
AZUREAI_OPENAI_BASE_URL=https://ft-test-sweden.openai.azure.com
AZUREAI_OPENAI_API_KEY=<snip>
```

which gets this which I assume is ~the same issue.

```
│ NotFoundError: Error code: 404 - {'error': {'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. If you created the deployment within │
│ the last 5 minutes, please wait a moment and try again.'}}   
```


(specifically the command I'm running is `inspect eval-set eval.py  --log-dir eval-set-logs --model=openai/azure/o3-mini`)

Okay, we unfortunately don't have an AzureAI deployment handy to test with right now. You might be able to help us out though -- I will say that 404 errors are super common in AzureAI when you don't get every last detail right (including sometimes slashes in URLs!). I would take your existing standalone code as a baseline and then use a debugger or print debugging to see exactly what differs when Inspect creates the client and calls the model. 

Ok, I hacked my local version of the OpenAI library - the only difference I see (other than Inspect uses `AsyncAzureOpenAI` and the script uses `AzureOpenAI`) is that the `base_url` is being passed through different, i.e.

working script - `base_url=https://ft-test-sweden.openai.azure.com/openai`

Inspect - `base_url=https://ft-test-sweden.openai.azure.com/openai/deployments/o3-mini`

Hmm, but I then hacked AsyncAzureOpenAI so both have

```
            organization=None,
            project=None,
            base_url=https://ft-test-sweden.openai.azure.com/openai,
            timeout=NOT_GIVEN,
            max_retries=2,
            default_headers=None,
            default_query={'api-version': '2024-12-01-preview'},
            http_client=None,
            websocket_base_url=None,
            _strict_response_validation=False,
            self._api_version = 2024-12-01-preview
        self._azure_ad_token = None
        self._azure_ad_token_provider = None
```
and that still isn't working. So there's something else missing

My brief attempts to get Inspect to use the non-await version failed; I'll have a think about this in the morning

@jjallaire is this the sort of thing you were expecting? Is there something else I should be looking for?

Alternatively, it would be great if we could hop on a call to take a look live; I'm typically free 10am-2am GMT

Finally got an Azure AI Foundry account setup so I can test all of this stuff. I did discover that there was cross-talk between OpenAI API keys and Azure API keys (which would use the OpenAI api key if defined). I fixed this as well as tested and updated docs on the configurations I tested here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1535

All mentions of Azure AI on this page now reflect the Azure AI Foundry deployments that I successfully tested with: https://inspect.aisi.org.uk/providers.html. For OpenAI this was:

``` bash
export AZUREAI_OPENAI_API_KEY=your-api-key
export AZUREAI_OPENAI_BASE_URL=https://your-url-at.azure.com
export AZUREAI_OPENAI_API_VERSION=2025-02-01-preview
inspect eval math.py --model openai/azure/gpt-4o-mini
```

One absolutely maddening thing about Azure is that the base URL used seems to change over time and ends up varying with different model types. For OpenAI on Azure AI Foundry I found that no "/models" suffix was required (or even supported) whereas when using Mistral or Llama models you did include the "/models" suffix.


Note that these fixes (as noted in the docs) are only on `main` not PyPI right now. To install from `main`:

```bash
pip install git+https://github.com/UKGovernmentBEIS/inspect_ai
```