Repository: promptfoo/promptfoo
Issue: `o1-preview` not working as expected: no output
Root cause: Configuration Error

**Describe the bug**
I tried setting up my evaluations as [instructed here](https://www.promptfoo.dev/docs/guides/gpt-vs-o1/) and have been  getting no output from `o1-preview`. My `config.yaml` looks like:

```
description: "Testing o1-preview"
prompts:
  - file://src/prompts.py:regular_prompt
  - file://src/prompts.py:augmented_prompt
providers:
  - id: openai:chat:gpt-4o-2024-08-06
    label: base
  - id: openai:o1-preview
    label: o1-preview
```
*I tried running this by using `openai:chat:o1-preview` and this ran but gave me the same output. I tested this using `o1-mini` and obtained the same results.

**To Reproduce**
With this set up, I get reasonable outputs from `base` but get empty outputs for `o1-preview`. Instaed of an output, I only get te assertion output strings such as `[BAD_FORMATTING] Error  JSON.`

**Expected behavior**
I expected the `o1-preview` model to output reasonable responses where nothing was output.

**Screenshots**
![image](https://github.com/user-attachments/assets/e3e5b3cf-87cf-405c-a7d0-3719693db620)
**System information (please complete the following information):**

- Promptfoo version:
0.89.4

When running evaluations using `openai:o1-preview` I get this warning, suggesting it should be at least `openai:chat:o1-preview`.

```
Unknown OpenAI model type: o1-preview. Treating it as a chat model. Use one of the following providers: openai:chat:<model name>, openai:completion:<model name>, openai:embeddings:<model name>, openai:image:<model name>
Unknown OpenAI model type: o1-mini. Treating it as a chat model. Use one of the following providers: openai:chat:<model name>, openai:completion:<model name>, openai:embeddings:<model name>, openai:image:<model name>
```

This means the example [here](https://www.promptfoo.dev/docs/guides/gpt-vs-o1/)  is incorrect since it is showing calling `o1-preview` as `openai:o1-preview`.

Hi @diego-delarue-ai, thanks for bringing this up! I just tried running the example with `o1-preview`, and it's working on my end. Could you please verify that you have access to the `o1-preview` model? According to [OpenAI's documentation](https://platform.openai.com/docs/guides/reasoning):

> 🧪 **o1 models are currently in beta**
>
> The o1 models are currently in beta with limited features. Access is limited to developers in [tier 5](https://platform.openai.com/docs/guides/rate-limits/usage-tiers?context=tier-five) (check your usage tier [here](https://platform.openai.com/settings/organization/limits)), with low rate limits. We are working on adding more features, increasing [rate limits](https://platform.openai.com/docs/guides/rate-limits), and expanding access to more developers in the coming weeks!

If you don't have access, that might explain the empty outputs and warnings you're encountering. Let me know if that's the case, and I'll see if there's anything else we can do to help.

Hey, @mldangelo, thanks for the quick reply!

I have a usage tier of 5 and am able to run the model perfectly fine when doing the following, so I don't think the issue is having access.

```
    completion = client.beta.chat.completions.parse(
        model="o1-preview",
        messages=messages,
    )
```

Right now that I tried running it with `openai:chat:o1-preview` I got this error as my output. Again, I'll note that I ran `openai:chat:gpt-4o-2024-08-06` alongside this and it worked fine.

```
API error: Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead., Type: invalid_request_error, Code: unsupported_parameter

{
  "error": {
    "message": "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.",
    "type": "invalid_request_error",
    "param": "max_tokens",
    "code": "unsupported_parameter"
  }
}
```

Some notes I will make about this evaluation:
- the prompt is input as a list of messages where each has a `role` and `content` field
- these prompts are generated by a python file
- `o1-preview` prompts do not include `system` message since this is not currently supported
- `gpt-4o-2024-08-06`, on the other hand, is given a `system` message 


Can you please double check your version with `promptfoo --version`? 

I am able to replicate your error on `0.87.0`. I can't replicate on the latest version. If that still doesn't work please throw some time on my calendar and we can look together: https://cal.com/michael-dangelo/15min

Okay, Ignore the `API error`, that was me just running in the wrong version, you were right. I am back to getting the original issue where nothing is output. I'll add some time to your calendar to talk about this more, thanks!

Hey @diego-delarue-ai, we figured out the cause of this error and @mldangelo fixed it here https://github.com/promptfoo/promptfoo/commit/b6975580c933b0b73950e90227a9a2855e124937

This fix will ship in the next release.