Repository: UKGovernmentBEIS/inspect_ai
Issue: Model identification insufficient for Azure Deployments
Root cause: Validation Gap

When OpenAI models are hosted on Azure, they're named `azure/openai/<configurable_deployment_name>`. This seems to break the assumptions made around naming in the OpenAI provider. In that code, the model name is stripped to the first slash and then used to identify which the model, but for the Azure deployments, this means the name is set to `openai/<configurable_deployment_name>`. As a result, the methods to check which model (or class of models) are unreliable. In #1462 I fixed the check for the `is_o_series` method so it will work correctly so long as a deployment name starts with `o`, but the problem is larger than this - the other checks in this file will fail for Azure regardless of deployment names.

An easy fix would be to update the other helper methods to behave like the update `is_o_series` method so they check the deployment name by looking after the last `/`. But a better fix would be to find a way to identify the actual model in use via Azure instead of just the deployment name. I'm not sure how to do that, so I'm just opening this issue instead of a PR.

https://github.com/UKGovernmentBEIS/inspect_ai/blob/746d772b380093c0553a2ab8856d277df8c01d16/src/inspect_ai/model/_openai.py#L54-L67

I believe that this is the correct fix (including reverting the original fix): https://github.com/UKGovernmentBEIS/inspect_ai/pull/1519

We don't currently have an Azure deployment to test on -- could you test this in your environment and report back to make sure we've gotten this right?


I can give it a try, but I'll point out that this fix still requires users to name their deployments such that they begin with the corresponding OpenAI model names. It's not an unreasonable requirement but if there's a way to avoid that it might be nice, otherwise perhaps worth documenting somewhere.

Interesting, I don't think that will be natural or obvious to people (and often devs aren't in control of deployment names if they are done by an ops team). I wonder if we just want to have all of the o-series checks be more liberal? (just look for o\d anywhere in the string). This could theoretically over-detect but I think failure to detect is going to be a much bigger problem.

I think there should notionally be a way to map from deployment name to the underlying model name through the AsyncAzureOpenAI interface. But in practice, I wasn't able to find anything that exposed this info in [their code](https://github.com/openai/openai-python/blob/6d0ecdd8ecbface903cf93c7571398b90b803b0b/src/openai/lib/azure.py#L88). The upstream Microsoft API has a ["list" API](https://learn.microsoft.com/en-us/rest/api/aiservices/accountmanagement/deployments/list?view=rest-aiservices-accountmanagement-2024-10-01&tabs=HTTP) which seems like it would include this info, but I couldn't figure out how to hit it through the AsyncAzureOpenAI interface.

Here's a change that does more liberal name matching for o1-mini and o1-preview, and more liberal name matching for o-series but only when "gpt" is not in the model name: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1519/commits/88fc812d68ae0324cea4b3f4503c31096f9198a2

Would be great if you could test this to make sure it works for your o-series deployment(s). I'll wait to merge until I hear from you.

This looks good! I see an error with o1-preview around an unknown `reasoning_effort` parameter being sent to the API but think that's an unrelated issue to this change (I see it with or without azure)

Okay, I added a change for the o1-preview issues as well: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1519/commits/068d2a84bf0b70b7259b7a63f69a84a420c907d9

Thanks so much for your help w/ this!!!

resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1519