Repository: UKGovernmentBEIS/inspect_ai
Issue: Feature Request: support for arbitrary provider adhering to known API format
Root cause: Unimplemented Feature Gap

Many different independent providers, like the currently supported together.ai, offer a similar pattern where you can send an openai-formatted request to a url along with a Bearer token, and get an openai API formatted response. Would be great to not have to hardcode each new provider, but rather just allow specifying a url, key_name (for env variable like <name>_API_KEY) and API-style  (e.g. openai styke).
This would open up using a wide variety of providers with specialized model versions not available elsewhere. For instance, hyperbolic with R1-Zero, or Huggingface API with Cerebras LLama 70B (2k tok/sec !).

I'm aware that I can go to an arbitrary url using --model-base-url <my-url> --model openai/mymodel  and setting my OPENAI_API_KEY to the api key I actually want to use. This is kinda obnoxious though. I would like to just have my OPENAI_API_KEY actually apply to OpenAI. And what about when OpenAI changes their API to some new version? Seems like you should be able to just state these three things (model-base-url, api_style, provider_name). 

you can do this right now: `get_model("openai/<model-name>", base_url="...", api_key="...")`

I guess another thing we could do is make a new model provider named "openai-compatible" and then do `OPENAI_COMPATIBLE_BASE_URL`, etc. Is that more what you had in mind?

But we are running tasks that require multiple models, from multiple different providers. So each one needs a different API key and different base url. As well as actual openai models, so those need the OPENAI_API_KEY to just be the true OpenAI key.

I mean, the alternative is to just keep adding specific model providers for every new provider, but that seems like a lot of unnecessary hard-coding of a thing that keeps changing? Up to you, I suppose. Have you seen the new huggingface inference provider endpoint btw? https://huggingface.co/blog/inference-providers 

I am totally open to a concrete suggestion on how we should change our interface here! We can do this now from Python:

```python
model = get_model("openai/<model-name>", base_url="...", api_key="...")
```

Are you proposing something like `openai-api/hyperbolic/r1` and then we resolve `HYPERBOLIC_MODEL_BASE_URL` and `HYPERBOLIC_API_KEY`?


Yes, that's basically what I had in mind. 
A confusing aspect of this is the question of whether to use the openai python package, or just compose a raw https request. Probably it makes sense to use the python package, and just make a note that people should remove the '/v1/chat/completions' part of the url in order to get the 'base url'?


If you use 'base url' according to the open_ai python package:

from openai import OpenAI

client = OpenAI(
	base_url="https://router.huggingface.co/novita",
	api_key="hf_xxxxxxxxxxxxxxxxxxxxxxxx"
)

messages = [
	{
		"role": "user",
		"content": "What is the capital of France?"
	}
]

completion = client.chat.completions.create(
	model="deepseek/deepseek-r1-distill-qwen-14b", 
	messages=messages, 
	max_tokens=500,
)

print(completion.choices[0].message)



If you use 'url' as if you were just composing a request, then you include the '/v1/chat/completions' as part of it:

curl 'https://router.huggingface.co/novita/v1/chat/completions' \
-H 'Authorization: Bearer hf_xxxxxxxxxxxxxxxxxxxxxxxx' \
-H 'Content-Type: application/json' \
--data '{
    "model": "deepseek/deepseek-r1-distill-qwen-14b",
    "messages": [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	],
    "max_tokens": 500,
    "stream": false
}'

Okay, we will create an `openai-api` provider that works exactly as described above. We'll be using the Python package so the base URL will _not_ include `/v1/chat/completions` (we'll make all of that clear in the docs)

This is now merged to `main`, docs here: https://github.com/UKGovernmentBEIS/inspect_ai/issues/1467