Repository: UKGovernmentBEIS/inspect_ai
Issue: For o1 models, Inspect sets `max_completion_tokens` equal to `max_tokens`
Root cause: Interface Contract Mismatch

In the OpenAI provider:
 https://github.com/UKGovernmentBEIS/inspect_ai/blob/dd1b33861c00e5014d6a6d1b9d18ec4e1d154c86/src/inspect_ai/model/_providers/openai.py#L267-L268

I find this to be highly unexpected behaviour. If you try to set `max_tokens` on an o1-series model, OpenAI raises a:

> openai.BadRequestError: Error code: 400 - {'error': {'message': "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}

I think it would be preferable to fail loudly and raise this, rather than silently converting `max_tokens` to `max_completion_tokens`. And add a `max_completion_tokens` parameter to Inspect. 

(For context, this behaviour has cost us thousands of dollars in wasted API credits. Because o1 is quite expensive and we're using Inspect to run relatively large evals. In fairness, if I had run our o1 evals at a smaller scale first, and checked the results manually in detail, I would have noticed sooner).

This is what OpenAI says about the two parameters (my emphasis):

> In previous models, the max_tokens parameter controlled both the number of tokens generated and the number of tokens visible to the user, which were always equal. However, with the o1 series, the total tokens generated can exceed the number of visible tokens due to the internal reasoning tokens.
> 
> Because some applications might rely on max_tokens matching the number of tokens received from the API, the o1 series introduces max_completion_tokens to explicitly control the total number of tokens generated by the model, including both reasoning and visible completion tokens. **This explicit opt-in ensures no existing applications break when using the new models.** The max_tokens parameter continues to function as before for all previous models.

https://platform.openai.com/docs/guides/reasoning

Those docs may be out of date. They explicitly say here that `max_tokens` is deprecated in favor of `max_completion_tokens`: https://platform.openai.com/docs/api-reference/chat/create#chat-create-max_tokens

They have chosen to introduce the new option for exactly the scenario that bit you: they want no surprises or implicit opt-in to additional reasoning cost. 

I think this does put us in a tough spot -- generally people want to set a single value of `max_tokens` rather than reason about "for openai models, I'll set `max_completion_tokens`, for other models, I'll set `max_tokens`. By deprecating, OpenAI is saying that they are not supporting _two distinct concepts_ in this realm, but rather doing something tactical so customers can avoid unexpected cost overruns. If we introduce separate concepts we are really tacking away from the actual APIs, creating permanent awkwardness and confusion just so we can play along with OpenAI's transition.

Our `GenerateConfig` is also an abstraction over many APIs so the field names won't always match exactly. There is still one concept in play and it just so happens that o1 models use a different parameter name. I don't want to permanently muddy the waters in the Inspect API to play along with their transition.