zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Is there a way to use a document as an input for an assertion ?
Root cause: Unimplemented Feature Gap

- description: "Simple  Assertions"
  vars:
    input_document: my_document.pdf
assert :

.
.
.

And use that as the input prompt for the agent

Support for PDF vars is added in #2218 !

We already support txt, json, js, and py var file formats.

Hi @SysOverdrive! I’m closing out this issue. Please pull the latest version of Promptfoo to load PDFs. Feel free to reopen this issue or reach out if you have any other questions or product feedback. Thanks!

This is amazing guys ! Thank you <3
Repository: promptfoo/promptfoo
Issue: azure openai doesn't work with ollamazure (a emulator of Azure OpenAI)
Root cause: Documentation Deficiency

**Describe the bug**

I am using [ollamazure](https://github.com/sinedied/ollamazure), which simulates the Azure OpenAI API (using Ollama). It works with all the tools I use except Promptfoo. When attempting to use it with Promptfoo, there is no readable error message except the following:

```
At least one evaluation failure occurred. This might be caused by the underlying call to the provider, or a test failure.
"id":"e5ea33f57a75d6ac88fa70e6a82943c08e0fa9811d161d67cb1da3c5ebf26746","provider":"azureopenai:chat:gpt-4o-mini","metrics":{"score":0,"testPassCount":0,"testFailCount":1,"assertPassCount":0,"assertFailCount":0,"totalLatencyMs":83238,"tokenUsage":{"total":0,"prompt":0,"completion":0,"cached":0,"numRequests":1},"namedScores":{},"namedScoresCount":{},"cost":0}}
```

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

.env file : 
```
# Ollamazure (Simulated Azure OpenAI with a local server)
OLLAMA_MODEL_NAME="phi3:3.8b-mini-4k-instruct-q4_K_M"
AZURE_OPENAI_DEPLOYMENT_NAME="gpt-4o-mini" # fake name
AZURE_OPENAI_API_KEY="123456" # fake api key
AZURE_OPENAI_ENDPOINT="http://localhost:4041" # real ollamazure endpoint
AZURE_OPENAI_API_VERSION="2024-10-01" # fake api version
AZURE_OPENAI_API_HOST="localhost:4041/"
```

run ollama, ollamazure and promptfoo 

promptfoo config : 
(second one is the model running directly in ollama, it works, first one is the simulated azureopenai that doesn't work
```
providers:
  - id: azureopenai:chat:gpt-4o-mini # env variables are in .env
  - id: ollama:chat:phi3:3.8b-mini-4k-instruct-q4_K_M # env variables are in .env
```



maybe it has something to do with the https and http ?
For example, i noticed that promptfoo needs `AZURE_OPENAI_API_HOST` without the https
however the other libraries use an endpoint with https 

do you process the https / http?

Try using `AZURE_API_BASE_URL` to set the full URL instead of just the host - that way you can use http instead of https!


Edit :
Thanks ! it works
 
i also have a problem reading the host from the yaml file. Using the apiHost in the config file doesn't work for me : 
```
providers:
  - id: azureopenai:chat:gpt-4o-mini 
    apiHost: xxxxxx.openai.azure.com/
```

```
[FAIL] Error: Azure API host   must be set.   
```

However, if works from the .env file : 
.env : 
`AZURE_API_HOST="xxxxxxxx.openai.azure.com/"
`



Should be

```yaml
providers:
  - id: azureopenai:chat:gpt-4o-mini 
    config:
      apiHost: xxxxxx.openai.azure.com/
```

Thank you

is it normal that the azureopenai also works with this en var  in promptfoo ? 

`AZURE_OPENAI_API_KEY` instead of `AZURE_API_KEY`
I tried both of them and they work with AZURE
Repository: promptfoo/promptfoo
Issue: Stateful Python provider - custom variables that last tests' lifetime
Root cause: Documentation Deficiency

Trying to write a custom Python script to test a chatbot I have in development. Quickly ran into an issue where the provider script is called anew for each test. The chatbot has an API which requires authentication. Upon login a conversation_id is generated that is to be passed in subsequent conversation messages.

I have an workaround where I login, generate the conversation_id and store it in a file. For subsequent requests, I read the conversation_id from the file and check if it is valid. If so, great, if not, start from login, write conversation_id to separate file. Rinse and repeat.

It would be great if there was a way to declare test realted variable somewhere with lifetime of entire test. That way any tokens, session related data etc. needed to run tests on a custom app can be generated once at the beginning of the tests and reused throughout the test.

Happy to consider other workarounds or existing features to accomplish this, in case I missed it. Also, happy to implement this feature and contribute to this awesome project. Is there a contribution guide? Thanks!

Hi @skrdvd!

Good news - **this is already supported!** Python providers use persistent worker processes where your module is loaded once and reused for all test cases within a single `promptfoo eval` run.

Your use case (authenticate once, reuse `conversation_id` across tests) works with module-level state:

```python
# stateful_chatbot_provider.py
_session_token = None
_conversation_id = None
_initialized = False

def _initialize_session():
    """Called once per evaluation session, not per test case."""
    global _session_token, _conversation_id, _initialized
    if _initialized:
        return

    # Authenticate and get conversation_id
    response = requests.post("https://api.chatbot.com/auth", ...)
    _session_token = response.json()["token"]

    response = requests.post("https://api.chatbot.com/conversations/new",
                            headers={"Authorization": f"Bearer {_session_token}"})
    _conversation_id = response.json()["conversation_id"]
    _initialized = True

def call_api(prompt, options, context):
    _initialize_session()  # Only runs once (lazy init)

    # Use persistent session state
    response = requests.post(
        "https://api.chatbot.com/messages",
        headers={"Authorization": f"Bearer {_session_token}"},
        json={"conversation_id": _conversation_id, "message": prompt}
    )
    return {"output": response.json()["reply"]}
```

**Key points:**
- Module-level variables persist across all test cases
- Default `workers: 1` ensures all requests hit the same worker with same state
- No file-based workarounds needed!

Documentation: https://www.promptfoo.dev/docs/providers/python/#worker-configuration

The file-based workaround you mentioned is no longer necessary. Closing as this functionality already exists - let us know if you run into any issues!
Repository: promptfoo/promptfoo
Issue: Cannot list multiple scenarios in separate yaml file
Root cause: Validation Gap

**Describe the bug**

I have the following in `promptfooconfig.yaml`:
```
scenarios:
 - file://scenarios.yaml
```
And the following in the `scenarios.yaml` file:
```
- config:
    - vars: ...
  tests:
    - vars: ...
- config:
    - vars: ...
  tests:
    - vars: ...
```
When running `eval` I get the following error:
```
> promptfoo eval
TestSuite Schema Validation Error:

  Validation error: Expected object, received array at "scenarios[0]"

Please review your promptfooconfig.yaml configuration.
node_modules/promptfoo/dist/src/evaluator.js:413
                for (const data of scenario.config) {
                                            ^

TypeError: scenario.config is not iterable
    at Evaluator.evaluate (node_modules/promptfoo/dist/src/evaluator.js:413:45)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async runEvaluation (node_modules/promptfoo/dist/src/commands/eval.js:189:9)
    at async doEval (node_modules/promptfoo/dist/src/commands/eval.js:352:5)

Node.js v20.18.0
```

**To Reproduce**
See above

**Expected behavior**
List multiple scenarios in the same yaml file.

**Screenshots**
N/A

**System information (please complete the following information):**

- Promptfoo version: `0.97.0`

**Additional context**
N/A

Hey @rvernica! Thanks for reporting this – sorry it took us a while to circle back!

Good news: this has been fixed! The issue was resolved in PR #4761 (v0.116.5+), which added proper array flattening when loading scenarios from external files.

## What was happening

When you used:
```yaml
scenarios:
  - file://scenarios.yaml
```

And `scenarios.yaml` contained an array of scenarios, the loader was returning a nested array `[[scenario1, scenario2]]` instead of `[scenario1, scenario2]`. This caused both the schema validation error and the `scenario.config is not iterable` TypeError you saw.

## The fix

We now flatten the scenarios array after loading from external files, so your original config should work as expected:

```yaml
# promptfooconfig.yaml
scenarios:
  - file://scenarios.yaml
```

```yaml
# scenarios.yaml
- config:
    - vars:
        foo: bar
  tests:
    - vars:
        baz: qux
- config:
    - vars:
        foo: bar2
  tests:
    - vars:
        baz: qux2
```

As a bonus, we also added glob pattern support in the same fix, so you can now do things like:
```yaml
scenarios:
  - file://scenarios/*.yaml
```

## Next steps

Just upgrade to v0.116.5 or later and you should be all set!
Repository: promptfoo/promptfoo
Issue: chore: error log on install
Root cause: Environment Incompatibility

Installing promptfoo 0.8.3 afresh (using pnpm, but npm should be the same):

```shell
src/cache.ts:4:26 - error TS7016: Could not find a declaration file for module 'cache-manager'. '/Users/username/Personal/Side Projects/2023/promptfoo/node_modules/.pnpm/cache-manager@4.1.0/node_modules/cache-manager/index.js' implicitly has an 'any' type.
  If the 'cache-manager' package actually exposes this module, consider sending a pull request to amend 'https://github.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/cache-manager'

4 import cacheManager from 'cache-manager';
```
Repository: promptfoo/promptfoo
Issue: Similarity check via CSV not working
Root cause: Validation Gap

 When running a similarity check via the CSV __expected column, I get the following error: 
 
```
[FAIL] Error: Invariant failed: Provider id() {
return `openai:${this.modelName}`;
} must implement callEmbeddingApi for similarity check
```

Using `openai:gpt-4`

I raised the same issue in this [ticket](https://github.com/promptfoo/promptfoo/issues/195#issuecomment-1736495303) which I thought I'd fixed, but I just got it to default to an equality check which would always fail.

Here is a simplified version of my CSV:

```
var1,var2,var3,__expected
Test1,Test2,Test3,similar(0.8):hello world
```

Running the same assertion in JS with the same data works fine.

Are you able to share your `providers` config?  Is it just `providers: openai:gpt-4`?

Yep that’s it

So I discovered it only throws this error if `defaultTest` includes an assertion that specifies another provider, like the following:

```Typescript
 defaultTest: [{
    type: "llm-rubric",
    value: "Are questions formatted as 3 numbered points on separate lines? Yes or No?",
    provider: "openai:gpt-3.5-turbo-0613",
  }]
```

If I don't specify a provider in the above assertion then everything works fine.

Hi @AZielinsky95,

Thanks for the report and for pinpointing the issue!  fb4b2437dee46ccf8e7ef83fdf8cfeb62abb9b7d should fix this issue by falling back to a model that supports embeddings.

This could also be resolved in config by setting `provider: openai:embedding:text-embedding-ada-002` on the assertion object.  But this is not possible in the CSV `__expected` format.
Repository: promptfoo/promptfoo
Issue: promptfoo redteam setup is obsolete
Root cause: Documentation Deficiency

**Describe the bug**
If i am not mistaken, these pages need to be updated : 
- https://www.promptfoo.dev/docs/red-team/quickstart/
- https://www.promptfoo.dev/docs/usage/command-line/#promptfoo-redteam-setup

It says to use `promptfoo redteam setup` but it doesn't exist in the commands of promptfoo redteam.
The available commands are : 

```
Commands:
  init [options] [directory]       Initialize red teaming project
  eval [options]                   Evaluate prompts
  generate [options]               Generate adversarial test cases
  run [options]                    Run red teaming process (init, generate, and evaluate)
  poison [options] <documents...>  Generate poisoned documents for RAG testing
  report [options] [directory]     Start browser ui and open to report
  plugins [options]                List all available plugins
  help [command]                   display help for command
```



@AmineDjeghri Sorry about this. We updated the docs yesterday before issuing a new release. I just released [0.98.0](https://github.com/promptfoo/promptfoo/releases/tag/0.98.0), which includes the updated commands mentioned in the docs. Everything should now align as expected. Please let us know if you encounter any further issues!

@mldangelo 

I think you should also update the docs of Azure OpenAI

The env variables can either be written like : `AZURE_KEYNAME` or `AZURE_OPENAI_KEYNAME`

https://github.com/promptfoo/promptfoo/blob/main/src/providers/azure.ts#L219

The AZURE_OPENAI ones are considered legacy which is why we removed them from the docs. Please use the AZURE_ instead. 

What do you mean by legacy ? The problem that I think of is the confusion for the projects that use other azure services. Having explicitly a variable named AZURE_OPENAI_API_KEY separates it from AZURE_AI_SEARCH_API_KEY for example.  
Are you planning to remove them?


Microsoft now supports a broader range of models, including non-OpenAI models like phi. To align with this shift and make the environment variable names more versatile, we removed “OPENAI” from the names. This change allows generic environment variables to be used seamlessly across different Azure services without being tied to a specific model provider. For example, AZURE_API_KEY now serves a broader purpose, while AZURE_OPENAI_API_KEY was more narrowly defined. While we have no immediate plans to remove the legacy AZURE_OPENAI_ variables and the legacy variables will continue to function for the foreseeable future (possibly years). As for other services, we don’t support AZURE_AI_SEARCH, but we’ll address it if and when we need to.
Repository: promptfoo/promptfoo
Issue: Link to google sheet not working
Root cause: External Dependency Breakage

The sheet is visible to "anyone with the link", however, running the tests silently fails with no variables getting loaded.

I'm using the SDK with Typescript. Here is my test suite with the sheet from your examples. (Also not working for me)
```Typescript
return {
    description: "Test Prompt A",
    providers: "openai:gpt-4",
    prompts: [prompt],
    tests: "https://docs.google.com/spreadsheets/d/1eqFnv1vzkPvS7zG-mYsqNDwOzvSaiIAsKB3zKg9H18c/edit?usp=sharing",
    defaultTest: tests,
    outputPath: "./src/tests/latestResults.json"
} as EvaluateTestSuite
```

Works fine when I use a local CSV.

This should be fixed by #222, which will be included in the next release.
Repository: promptfoo/promptfoo
Issue: Add openai model name parameter in openai provider's config
Root cause: Documentation Deficiency

Is it possible to add a parameter so we can change and load openai model name from a environment variable please ?


Also, for Azure OpenAI, i think the `AZURE_API_VERSION` is missing : https://www.promptfoo.dev/docs/providers/azure/

Re. loading model names, this should be supported now. You can try with `{{ env.YOUR_ENV_VAR_NAME }}`. For example `openai:chat:{{env.MODEL_NAME}}`

For AZURE_API_VERSION  we would gladly accept a PR to add support. If you are interested in helping add support, please take a look here: https://github.com/promptfoo/promptfoo/blob/main/src/providers/azure.ts#L44 and https://github.com/promptfoo/promptfoo/blob/main/src/types/env.ts#L12 

Thanks! 

Thanks, it works
however i think there is a problem with the render of the name 😅
![image](https://github.com/user-attachments/assets/19ea1894-e8ae-4470-af8d-d4137357b121)


label attributes are rendered https://github.com/promptfoo/promptfoo/blob/main/src/providers.ts#L519-L520 but the id is not. We would gladly accept a PR if this is something you are interested in contributing. 

As a workaround you can set 

```yaml
- id: openai:chat:{{env.MODEL_NAME}}
  label: openai:chat:{{env.MODEL_NAME}}
```

as your provider 

will this work ? 


// ... other code ...
ret.label ||= getNunjucksEngine().renderString(String(options.label || ''), {});
ret.id ||= getNunjucksEngine().renderString(String(options.id || ''), {});


Hi @AmineDjeghri,

Thanks for this question! Good news - this feature already works in promptfoo.

You can use template syntax to load model names from environment variables:

```yaml
providers:
  - id: openai:chat:{{ env.MODEL_NAME }}
```

Or if you want the name to display correctly in the UI:
```yaml
providers:
  - id: openai:chat:{{ env.MODEL_NAME }}
    label: openai:chat:{{ env.MODEL_NAME }}
```

Then set your environment variable:
```bash
export MODEL_NAME=gpt-4
promptfoo eval
```

This works for OpenAI and all other providers. There's a minor UI display issue where the `id` doesn't render templates by default, but the workaround above (setting both `id` and `label`) handles it.

If you'd like to contribute a fix for the UI rendering, the code is in [`src/providers.ts:519-520`](https://github.com/promptfoo/promptfoo/blob/main/src/providers.ts#L519-L520).

See the [OpenAI Provider documentation](https://www.promptfoo.dev/docs/providers/openai) for more info.

Thanks for asking!
Repository: promptfoo/promptfoo
Issue: Bug: `--no-cache` leads to no embeddings found
Root cause: Others

I have a `promptfooconfig.yaml` with `promptfoo==0.25.0` that looks like this:

```yaml
---
defaultTest:
    options:
        provider: localai:embeddings:bert-embeddings
prompts: [prompts.txt]
providers: ["ollama:llama2:13b"]
sharing: false
tests:
    - vars:
          area: foo
          result: bar
      assert:
          - type: similar
            value: {{ result }} <redacted> {{ area }}.
```

Running `promptfoo eval --no-cache`, I get:

```
[FAIL] API response error: Error: No embedding found in LocalAI embeddings API response: {"error":{"code":500,"message":"rpc error: code = Unavailable desc = error reading from server: EOF","type":""}}
```

I think somehow the `--no-cache` option is preventing `promptfoo` from using LocalAI embeddings.  It seems there's still a dependency on cache entries to pass data.

I'm having trouble reproducing this issue with the config you provided.  My steps are:
- clear the cache `promptfoo cache clear` (or delete `~/.promptfoo` manually)
- run `promptfoo eval` normally to fill the cache
- run `promptfoo eval --no-cache` to make sure it doesn't use the cache

Any more detail on your end I should be paying attention to?

What I was seeing yesterday was:

- Usage without `--no-cache`: worked fine
- Usage with `--no-cache`: failed with the original post's error

However, today I can't repro it either 😵 .  Going to close this out, sorry for the churn
Repository: promptfoo/promptfoo
Issue: Error formatting and structure lost in the UI when using Custom Python Provider
Root cause: Validation Gap

**Describe the bug**
I have a custom Python provider. The provider reports errors as specified in the [documentation](https://www.promptfoo.dev/docs/providers/python/#python-script) The errors contain structure and indentation but this gets lost in the web UI.

Here is how the error appears in the terminal:
![image](https://github.com/user-attachments/assets/39448adf-db08-4a89-8a79-318b77fc8ac1)

Here is how the error appears in the web UI
![image](https://github.com/user-attachments/assets/eed2ea8e-ce63-434c-ac69-5433fb7ba795)

Notice how the newlines and indentation are lost.

**To Reproduce**
Write a custom Python provider that returns an error with structure and indentation.

**Expected behavior**
Preserve the formatting or provide a way to format the error message for the web UI.

**Screenshots**
See the description above.

**System information:**

- Promptfoo version: `0.97.0`

**Additional context**
N/A

@rvernica Thanks for reporting this issue! This has been fixed in version [0.100.6](https://github.com/promptfoo/promptfoo/releases/tag/0.100.6). Please upgrade to the latest version and let me know if you see any other. Thanks for helping us improve promptfoo! 🙌
Repository: promptfoo/promptfoo
Issue: Changin the target in redteam doesn't do anything
Root cause: Configuration Error

**Describe the bug**
In the redteam feature, changing the target doesn't seem to work. 
Maybe i'm missing something


In the yaml file : 
```
  - id: https
    label: 'travel-agent-agent'
    config:
      url: 'https://example.com/generate'
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
      body:
        myPrompt: '{{prompt}}'
```
or 

`  - id: azureopenai:chat:gpt-4o-mini
`
result : 
```
Error in extraction: API error: 404 Not Found
{"error":{"message":"model \"gpt-4o-2024-05-13\" not found, try pulling it first","type":"api_error","param":null,"code":null}}
An unexpected error occurred: Failed to perform extraction: API error: 404 Not Found
{"error":{"message":"model \"gpt-4o-2024-05-13\" not found, try pulling it first","type":"api_error","param":null,"code":null}}

```



- Promptfoo version: 0.99



after further investigation, having the following env variable set, causes the problem:

```
OPENAI_API_KEY="t"
```
commenting it from my .env file solves the problem. 

- maybe the plugins llm graders use gpt-4o-2024-05-13 with this key ?
- also, can we override the plugin llm grader ? 


The attacker model is different from the target, and defaults to GPT-4o.

See here for info on how to change the attacker model:
https://www.promptfoo.dev/docs/red-team/configuration/#providers

Or, just unset OPENAI_API_KEY and it will use a remote provider.

On Tue, Nov 26, 2024 at 6:40 AM Amine Djeghri ***@***.***>
wrote:

> after further investigation, having the following env variable set, causes
> the problem:
>
> OPENAI_API_KEY="t"
>
> commenting it from my .env file solves the problem.
>
>    - maybe the plugins llm graders use gpt-4o-2024-05-13 with this key ?
>    - also, can we override the plugin llm grader ?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/2245#issuecomment-2500983052>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJW2Q6IVBZE2GMPX3OD2CSB33AVCNFSM6AAAAABSQRTXYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBQHE4DGMBVGI>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>