Repository: promptfoo/promptfoo
Issue: Support RBAC for Azure OpenAI (Key based authentication is disabled for this resource)
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
At Microsoft, we are moving away from key-based authentication and towards RBAC. As such, we cannot use AZURE_OPENAI_API_KEY on our resources.

When we try:

`npx promptfoo eval --verbose` on the [`azure-openai`](https://github.com/promptfoo/promptfoo/tree/0.95.0/examples/azure-openai) example with promptfoo@0.95.0

we get:

"Azure OpenAI API response: {"error":{"code":"AuthenticationTypeDisabled","message":"Key based authentication is disabled for this resource."}}" 

**Describe the solution you'd like**
If there was some way to use [`@azure/msal-node`](https://github.com/AzureAD/microsoft-authentication-library-for-js/tree/dev/lib/msal-node) or similar to authenticate the principal (user or pipeline agent) in the relevant Azure tenant, then use their Azure credentials to connect to the resource, that'd be great.

**Describe alternatives you've considered**
My team is actively working on wrapping [`node-package-typescript`](https://github.com/promptfoo/promptfoo/tree/0.95.0/examples/node-package-typescript) with RBAC auth logic via `@azure/msal-node` or something similar to workaround this issue.

**Additional context**
We aren't experts at this--there may be a better way, and we'll update with what we find :)

Related: #1074 

Ref [msal-node sample](https://github.com/AzureAD/microsoft-authentication-library-for-js/blob/dev/extensions/samples/msal-node-extensions/index.js)

Thanks for bringing this up.  Definitely seems like something we should bake into our Azure provider https://github.com/promptfoo/promptfoo/blob/main/src/providers/azureopenai.ts

@mwiemer-microsoft what's your ideal flow here?  Are you suggesting that Promptfoo provides a login endpoint?

@typpo I've been trying to figure that out. For reference, our C# code can use Visual Studio credentials, and our deployed web apps can use browser credentials, but of course neither of those are possible with this isolated process. Ideally, the promptfoo process is able to fetch some environment variables, including a login endpoint, silently make a "get token" request, and use the token to make requests against the LLM directly.

Slightly less ideal, an API to provide a custom login function would be great, as that would prevent us from having to manually ping the LLM as well.

Simplest, accepting an `AZURE_OPENAI_BEARER_TOKEN=Bearer eyJ0...` environment variable in place of the `AZURE_OPENAI_API_KEY` would work great for us. We are able to manually scrape this token from Azure OpenAI Playground every 4 hours, and we're currently providing it to a custom "pingLLM" function.

Here is our current code:

```ts
// request.ts (full file)
import dotenv from 'dotenv';
import fetch from 'node-fetch';

/** Simplified interface of response body JSON for parsing */
interface GPT4oResponse {
    choices?: {
        message?: {
            content?: string;
        };
    }[];
}

// Load environment variables from .env file
dotenv.config();

/**
 * Pings the GPT-4o UAT deployed model in Azure OpenAI and returns the LLM response (empty string if response was filtered).
 * Throws error is BEARER_TOKEN is not set in environment.
 *
 * @param body Stringified JSON payload to send to the model
 *
 * Requires a bearer token present in `.env` BEARER_TOKEN, see `.env.example`.
 */
export default async function pingLLM(body: string): Promise<string> {
    /**
     * See readme for details
     * @example 'Bearer eyJ0...'
     */
    const bearerToken = process.env.BEARER_TOKEN;
    if (!bearerToken) {
        throw new Error(
            'BEARER_TOKEN is not defined in the environment variables.',
        );
    }
    const deploymentName = 'GPT-4o'; //'gpt-4o-mini' // names defined by each team
    const url =
        'https://__ourOpenAIResource__.openai.azure.com/openai/deployments/' +
        deploymentName +
        '/chat/completions?api-version=2024-10-01-preview';
    const response = await fetch(url, {
        headers: {
            accept: '*/*',
            authorization: bearerToken,
            'content-type': 'application/json',
        },
        body: body,
        method: 'POST',
    });
    const responseJson = (await response.json()) as GPT4oResponse;
    // console.log('HTTPS response:\n', response);
    // console.log('Response body:\n', JSON.stringify(responseJson));
    return responseJson?.choices?.[0]?.message?.content ?? '';
}

// main.ts (snippet)
const providers: EvaluateTestSuite['providers'] = [
    async (prompt, context) => {
        const requestBody = {
            messages: [
                {
                    role: 'system',
                    content:
                        'You are an AI assistant that helps people find information.',
                },
                { role: 'user', content: prompt },
            ],
            // copied from the relevant config.json file (currently only one prompt file tested)
            // todo find a way to parse JSON based on the prompt file path
            max_tokens: 500,
            temperature: 0.0,
            top_p: 0.0,
            presence_penalty: 0.0,
            frequency_penalty: 0.0,
            // additional properties
            stop: null,
            stream: false, // we want the whole result at once
        };
        const response = await pingLLM(JSON.stringify(requestBody));
        console.log(logString(prompt, JSON.stringify(context?.vars), response)); // just debug logging, cut for GitHub
        return { output: response };
    },
];
```

Any way to reduce the boilerplate here would be great

Hi @mwiemer-microsoft and @typpo, 
We are also super interested in using promptfoo and we can only do so if we can get it to work without api keys. 

I had a stab at adding support for azure cli credentials in my PR: https://github.com/promptfoo/promptfoo/pull/2149 . A bit more work needs to be done in documentation and error messages but It is functional. You only need to supply the following in a config

```
    provider:
      id: azure:chat:gpt-4-deployment-name
      config:
        apiHost: 'xxxxxxx.openai.azure.com'
```

Would be great to get your feedback early on so I know I am on the right path. Thanks in advance 

@ali42, this is awesome to see, would definitely land this PR!

Hi @mwiemer-microsoft and @ali42, thank you for your help! I’m going to close this issue as I believe it should now be working as intended. If you encounter any further issues or have additional feedback, please feel free to comment here, and we’ll gladly reopen and investigate further. Appreciate the assist!