Repository: tatsu-lab/alpaca_eval
Issue: openai_configs.yaml when using Azure only
Root cause: Documentation Deficiency

Hi,

I wonder how to set up openai_configs.yaml when only using Azure OpenAI. Do I need to set up "default"?

I tried the following but it does not work:

```
gpt-4-1106-preview: # only when using `model_name: gpt-4-1106-preview`
    - "default" # this will append all the `default` configs

    - client_class: "openai.AzureOpenAI" # doesn't use the `openai.OpenAI` client class
      # the following are passed to the `AzureOpenAI` client class
      azure_deployment: "gpt-4-1106"
      api_key: "<your Azure OpenAI API key here>"
      azure_endpoint: "<your Azure OpenAI API base here>"
      api_version: "2023-07-01-preview"
```

The error shows: openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable. 

Thanks!

Hi @Yuancheng-Xu,

I currently don't have access to AzureOpenAI anymore but should have it next week to help you. In terms of you question and what I think the error is:

1. If you want to only use AzureOpenAI, then you shouldn't add "default". How you should read those configs is that it maps the model name to ano (ordered) list of client to use if the previous one hits a rate limit. "default" is the model-independent list. I'm assuming you didn't set the "default" configs here so as a result it tries to use `OPENAI_API_KEY ` which is not set. Long story short: remove "default" and it should work.

4. One issue you will face is that I changed the annotator from `"gpt-4-1106-preview"` to `"gpt-4-turbo"` today given the new stable OpenAI release. So if you are on the latest branch/version of alpaca_eval you should use the latter. I don't know whether Azure already supports the new release though. If it doesn't, then you should keep `"gpt-4-turbo"` as the config name, but use `"gpt-4-1106"` as the azure deployment model. It it does, then use the name of that `"gpt-4-turbo"`.

Putting it all together, your configs should look like this to only use Azure:

```
gpt-4-turbo: # on the latest alpaca_eval
    - client_class: "openai.AzureOpenAI" 
      azure_deployment: "gpt-4-1106" # if possible, replace that by the corresponding name of "gpt-4-turbo" in Azure
      api_version: "2024-03-01-preview" # if you change azure_deployment then you likely need to change this also
      api_key: "<your Azure OpenAI API key here>"
      azure_endpoint: "<your Azure OpenAI API base here>"
```

and `pip install -U alpaca_eval`

Let me know if that works!
      

Hi @YannDubs ,
Thank you for the prompt reply! 

I tried the configs you provided but the same error still occurred. I realized that I may have put the `openai_configs.yaml` in the wrong place. According to the doc, create `openai_configs.yaml` in the current directory. What does "Current directory" mean? 

What I did is I create `openai_configs.yaml` in a folder, and within the same folder I ran 
```
export IS_ALPACA_EVAL_2=False
alpaca_eval --annotators_config 'alpaca_eval_gpt4' --model_outputs xxx
```
Did I miss something here? Thank you!

@Yuancheng-Xu Current directory meant in `client_configs/`. You can change that by setting `export OPENAI_CLIENT_CONFIG_PATH=./openai_configs.yaml` to use a config in the path you are using. I'll improve the documentation for that.

Why are you using AlpacaEval 1? AlpacaEval 1 uses `gpt-4` as a model evaluator so your config should look like
```
gpt-4: # to use AlpacaEval 1
    - client_class: "openai.AzureOpenAI" 
      azure_deployment: "<the name of gpt-4 (0613) on azure>" 
      api_version: "<current azure API version>" 
      api_key: "<your Azure OpenAI API key here>"
      azure_endpoint: "<your Azure OpenAI API base here>"
```
I would highly recommend using AlpacaEval 2 which is cheaper, faster, and better. 

Save the above to `./openai_configs.yaml` and then use 
```
export OPENAI_CLIENT_CONFIG_PATH=./openai_configs.yaml
export IS_ALPACA_EVAL_2=False
alpaca_eval --model_outputs xxx
```

@YannDubs Thank you! After `export OPENAI_CLIENT_CONFIG_PATH=./openai_configs.yaml` it works!

I used AE 1 since I only have access to Azure and according to the doc, "Unfortunately it's not currently possible to use Azure with AlpacaEval2.0 as they do not provide logprobs, this will hopefully be fixed soon."

However I just checked and now azure supports logprobs in `2024-03-01-preview` API version . I use the following configs:

```
gpt-4-turbo: # on the latest alpaca_eval
    - client_class: "openai.AzureOpenAI" 
      azure_deployment: "gpt-4-1106-preview" 
      api_version: "2024-03-01-preview" 
      api_key: "<key>"
      azure_endpoint: "<endpoint>"
```

With the setup above, I am able to run AE2 with Azure. Thank you for your help!

Yes, Azure added logprobs very recently. I updated the README!

Thank you @YannDubs !

A follow up question: If I have access to two Azure endpoints, I wonder if the following configuration, based on my understanding, is correct.

```
gpt-4-turbo: # on the latest alpaca_eval
    - client_class: "openai.AzureOpenAI" 
      azure_deployment: "gpt-4-1106-preview" 
      api_version: "2024-03-01-preview" 
      api_key: "<key>"
      azure_endpoint: "<endpoint>"

    - client_class: "openai.AzureOpenAI" 
      azure_deployment: "gpt-4-1106-preview" 
      api_version: "2024-03-01-preview" 
      api_key: "<another key>"
      azure_endpoint: "<another endpoint>"
```

Yes, that should work! When you hit a rate limit it should switch client for you. Note that this means you can likely increase the number of concurrent calls (by default 5). E.g. try `export OPENAI_MAX_CONCURRENCY=10`.

Let me know if you have issues as I don't test thoroughly on Azure so I'm not sure the rate limit catching is still up to date, but hopefully they use the same as OpenAI client.