Repository: promptfoo/promptfoo
Issue: AzureOpenAI and ClientSecretCredential
Root cause: Configuration Error

Hi.
I need to use AzureOpenAI with ClientSecretCredential to authenticate.
then, I need to pass to the ClientSecretCredential a tenant_id, client_id, client_secret, authority and scope tp get a Token.
![image](https://github.com/promptfoo/promptfoo/assets/6631983/4f2de3bf-79c9-43ce-92a4-d4007ce33e18)

I also need to pass to the LLM the 
- deployment_name
- model_name
- openai_api_base
- openai_api_version
- openai_api_key
- openai_api_type
- temperature
- max_tokens

How I should proceed to setup my Azure provider in my YML? should i need to create a custom provider? is there any example passisng all this parameters?








Right now you would have to generate the token manually and pass it in the YML or as an environment variable, or use a [custom script](https://promptfoo.dev/docs/providers/custom-script) (python) or [custom ApiProvider](https://promptfoo.dev/docs/providers/custom-api) (javascript).

I'll look into how we can support ClientSecretCredential for Azure natively in promptfoo...

#389 adds support for Azure ClientSecretCredential flow

Support for ClientSecretCredential landed in 0.35.1

Hi @typpo 
hope you are doing fine!!
I am playing with ClientSecretCredential using promptfoo 0.35.1 version and I have some issues;

Based on my AvureOpenAI configuration, I need to pass a "openai_api_type = azure_ad" and a "openai_api_version=2023-09-01-preview". Are these two parameters setup in the Azure ClientSecretCredential flow implemented? I am not able to see them in #389 

Here you have my provider configuration:

```
providers:
  - id: azureopenai:chat:gpt4
    config:
      apiHost: cs-oai-XXXX-genai35dev.openai.azure.com
      azureTenantId: e0793d39-XXXX-XXXXX-b129-198edd916feb
      azureClientId: fb2bf190-33f1-XXXX-XXXXX-0dd94e106797
      azureClientSecret: hje8Q~XXXXX-RwHlTS2p-IILVh.t.XXXXXX
      deploymentName: csdep-XXXXXX-gpt-4
      deployment_id: gpt-4
      #openai_api_type="azure_ad", 
      #openai_api_version=2023-09-01-preview
      temperature: 0
      max_tokens: 50
```
And this is the error i have running the eval (where the preview is not the correct one, I need to pass "2023-09-01-preview") 

`API call error: Error: Request failed after 4 retries: request to https://xcs-oai-25730602-genai35dev.openai.azure.com/openai/deployments/gpt4/chat/completions?api-version=2023-07-01-preview failed, reason: getaddrinfo ENOTFOUND xcs-oai-25730602-genai35dev.openai.azure.com`

Thanks!

Thanks @typpo,  I have seen you are adding apiVersion; in #399 what about openai_api_type to pass"azure_ad"

Do you know if it's actually necessary to pass `azure_ad` to the API in any way?  In our current logic, the ClientCredentials behavior happens automatically if tenant id/client id are set.

Thanks @typpo for your quick response!!

According to the [documents](https://python.langchain.com/en/latest/modules/models/llms/integrations/azure_openai_example.html) of langchain, user can use Azure OpenAI service instead of origin OpenAI by setting some env config.

```
# Set this to `azure`
export OPENAI_API_TYPE=azure
# The API version you want to use: set this to `2022-12-01` for the released version.
export OPENAI_API_VERSION=2022-12-01
# The base URL for your Azure OpenAI resource.  You can find this in the Azure portal under your Azure OpenAI resource.
export OPENAI_API_BASE=https://your-resource-name.openai.azure.com
# The API key for your Azure OpenAI resource.  You can find this in the Azure portal under your Azure OpenAI resource.
export OPENAI_API_KEY=<your Azure OpenAI API key>
```

I have tested **removing the openai_api_type="azure_ad" from an example using langchain and if fails.** 

![image](https://github.com/promptfoo/promptfoo/assets/6631983/d610c674-9ce4-4071-b0bd-5568b3b4ded6)

```
Error: Command failed: python langchain_example.py Answer the following concisely: Write a poem about the sea.

- Think carefully & step-by-step.
- You must answer the question directly, without speculation.
- You cannot access realtime information. Consider whether the answer may have changed in the 2 years since your knowledge cutoff.
- If you are not confident in your answer, begin your response with "Unsure". {"id":"exec:python langchain_example.py","config":{"basePath":""}} {"vars":{"message":"Write a poem about the sea."}}
Traceback (most recent call last):
  File "C:\Users\XXXX\code\GenAIProductExpert_257306\promptfoo\langchain\langchain_example.py", line 34, in <module>
    print(llm(messages=[msg]))
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain_core\language_models\chat_models.py", line 632, in __call__
    generation = self.generate(
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain_core\language_models\chat_models.py", line 378, in generate
    raise e
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain_core\language_models\chat_models.py", line 368, in generate
    self._generate_with_cache(
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain_core\language_models\chat_models.py", line 524, in _generate_with_cache
    return self._generate(
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain_community\chat_models\openai.py", line 434, in _generate
    response = self.completion_with_retry(
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain_community\chat_models\openai.py", line 360, in completion_with_retry
    return _completion_with_retry(**kwargs)
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\tenacity\__init__.py", line 289, in wrapped_f
    return self(f, *args, **kw)
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\tenacity\__init__.py", line 379, in __call__
    do = self.iter(retry_state=retry_state)
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\tenacity\__init__.py", line 314, in iter
    return fut.result()
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 451, in result
    return self.__get_result()
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 403, in __get_result
    raise self._exception
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\tenacity\__init__.py", line 382, in __call__
    result = fn(*args, **kwargs)
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain_community\chat_models\openai.py", line 358, in _completion_with_retry
    return self.client.create(**kwargs)
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\openai\api_resources\chat_completion.py", line 25, in create
    return super().create(*args, **kwargs)
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\openai\api_resources\abstract\engine_api_resource.py", line 155, in create
    response, _, api_key = requestor.request(
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\openai\api_requestor.py", line 299, in request
    resp, got_stream = self._interpret_response(result, stream)
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\openai\api_requestor.py", line 710, in _interpret_response
    self._interpret_response_line(
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\openai\api_requestor.py", line 775, in _interpret_response_line
    raise self.handle_error_response(
openai.error.AuthenticationError: Access denied due to invalid subscription key or wrong API endpoint. Make sure to provide a valid key for an active subscription and use a correct regional API endpoint for your resource.


Error: Command failed: python langchain_example.py Answer the following concisely: Write a poem about the sea.

- Think carefully & step-by-step.
- You must answer the question directly, without speculation.
- You cannot access realtime information. Consider whether the answer may have changed in the 2 years since your knowledge cutoff.
- If you are not confident in your answer, begin your response with "Unsure". {"id":"exec:python langchain_example.py","config":{"basePath":""}} {"vars":{"message":"Write a poem about the sea."}}
Traceback (most recent call last):
  File "C:\Users\XXXX\code\GenAIProductExpert_257306\promptfoo\langchain\langchain_example.py", line 34, in <module>
    print(llm(messages=[msg]))
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain_core\language_models\chat_models.py", line 632, in __call__
    generation = self.generate(
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain_core\language_models\chat_models.py", line 378, in generate
    raise e
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain_core\language_models\chat_models.py", line 368, in generate
    self._generate_with_cache(
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain_core\language_models\chat_models.py", line 524, in _generate_with_cache
    return self._generate(
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain_community\chat_models\openai.py", line 434, in _generate
    response = self.completion_with_retry(
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain_community\chat_models\openai.py", line 360, in completion_with_retry
    return _completion_with_retry(**kwargs)
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\tenacity\__init__.py", line 289, in wrapped_f
    return self(f, *args, **kw)
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\tenacity\__init__.py", line 379, in __call__
    do = self.iter(retry_state=retry_state)
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\tenacity\__init__.py", line 314, in iter
    return fut.result()
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 451, in result
    return self.__get_result()
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 403, in __get_result
    raise self._exception
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\tenacity\__init__.py", line 382, in __call__
    result = fn(*args, **kwargs)
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain_community\chat_models\openai.py", line 358, in _completion_with_retry
    return self.client.create(**kwargs)
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\openai\api_resources\chat_completion.py", line 25, in create
    return super().create(*args, **kwargs)
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\openai\api_resources\abstract\engine_api_resource.py", line 155, in create
    response, _, api_key = requestor.request(
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\openai\api_requestor.py", line 299, in request
    resp, got_stream = self._interpret_response(result, stream)
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\openai\api_requestor.py", line 710, in _interpret_response
    self._interpret_response_line(
  File "C:\Users\XXXX\AppData\Local\Programs\Python\Python310\lib\site-packages\openai\api_requestor.py", line 775, in _interpret_response_line
    raise self.handle_error_response(
openai.error.AuthenticationError: Access denied due to invalid subscription key or wrong API endpoint. Make sure to provide a valid key for an active subscription and use a correct regional API endpoint for your resource.

    at ChildProcess.exithandler (node:child_process:422:12)
    at ChildProcess.emit (node:events:514:28)
    at maybeClose (node:internal/child_process:1105:16)
    at ChildProcess._handle.onexit (node:internal/child_process:305:5)
```


**Keeping the openai_api_type="azure_ad", it works.**
![image](https://github.com/promptfoo/promptfoo/assets/6631983/80fe1661-e8d9-4bb4-9de4-b4a4ecb9e0b7)

![image](https://github.com/promptfoo/promptfoo/assets/6631983/98bf331c-e7a0-4c24-a921-8c9902a951eb)




That looks to be a langchain parameter, not part of the Azure API.  Based on LangChain's usage it seems like our approach is ok: https://github.com/search?q=repo%3Alangchain-ai%2Flangchain%20%22azure_ad%22&type=code

Hi @typpo, I will try adding the apiVersion to the config parameter without the penai_api_type="azure_ad". 
I will do it once the new version will be published. Do you know when wil be published?
I let you know the result. 
Thanks again!!