zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: OpenAI Assistant Evalauation
Root cause: Documentation Deficiency

Hello, 
we are a team of 4 Student and are looking for a way to eval our openai assistant. We are not getting behind it how to check if the assistant is using our external files  which we passed to him or if the bot is hallucinating. We would love to get some help with that. When we want to use context-faithfulness what do we  have to pass to the question var ? 

Greetings Joe 

Hi @JohannesGlock, were you able to get what you needed?  There is a guide on evaluating OpenAI assistants: https://promptfoo.dev/docs/guides/evaluate-openai-assistants/ and you can also check out the RAG guide to learn how to use external files in the context: https://promptfoo.dev/docs/guides/evaluate-rag

Yes thanks a lot!! . But we are still on it because we are struggeling on how to include more context. There seems to be maximum words for context. As we want to get the context-faithfulness  😀
Repository: promptfoo/promptfoo
Issue: Support stricter config parsing & fail on no assertions
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I was working on existing evals that were "passing" because nothing was asserted. This is related to two problems:
- Promptfoo is fine with any keys in the config. I just tried running a config with `junk: useless` at the top level and it parsed and ran just fine.
- Misspelling anything in the config can lead to evals passing without asserting anything. In my case, the file I was working on had a top-level `assert` that wasn't nested under `defaultTest`.

**Describe the solution you'd like**
Two parts:
- Warn or fail on superfluous config keys. Failure can be based on a `--strict` flag.
- Add option to fail evals with no assertions (maybe based on the same `--strict` flag). 

**Describe alternatives you've considered**
We're currently using a wrapper script to work around these problems.
Repository: promptfoo/promptfoo
Issue: Feature Request: Export to csv/json
Root cause: Documentation Deficiency

As a workaround for #428 I'd like to export results from a specific run to JSON (e.g. in the UI) for easier analysis.

You should be able to get JSON directly from the eval command, e.g. with `promptfoo eval -o filename.json`.  This outputs the same file format that's used to populate the UI.
Repository: promptfoo/promptfoo
Issue: Question: OpenAI Assistant 
Root cause: Documentation Deficiency

To evaluate our OpenAI Assistant we included our context into the yaml file. But there is very low capacity of word we can use. You can pass around 32000 word into the context. This is not enough to evalaute the Assistant which uses a document with let´s say 200.000 word. 

Is there any workaround to get bigger context windows ? 

Thank you!


<img width="825" alt="Bildschirmfoto 2024-01-31 um 18 45 06" src="https://github.com/promptfoo/promptfoo/assets/117032423/0590c937-447c-4a8d-b620-e8da0a53e688">


Hi Johannes, the context window is a property of the model, it's not set by promptfoo.  You will have to adjust the context size to fit with whichever model you're using.

Closing this out for now - let me know if you run into any issues on the promptfoo side!
Repository: promptfoo/promptfoo
Issue: Promptfoo view (web viewer) does not show anymore latest results, while test is running
Root cause: Algorithmic Error

## Test Results Not Showing in Real-Time Web Viewer

### Description
In version 0.114.5, the web viewer behavior has changed regarding the display of test results. Currently, test results are only visible after the entire test run is completed. Previously, results were rendered in real-time as individual test cases finished.

### Steps to Reproduce
1. Start a test collection that runs multiple test cases
2. While tests are running, execute `promptfoo view`
3. Observe that no results are displayed until all tests complete

### Expected Behavior
- Test results should display in real-time as each individual test case completes
- Users should be able to view partial results while the test collection is still running

### Current Behavior
- Web viewer remains empty during test execution
- Results only appear after all tests in the collection have finished

### System Information
```json
{
  "version": "0.114.5",
  "platform": {
    "os": "win32",
    "release": "10.0.19045",
    "arch": "x64",
    "nodeVersion": "v22.14.0"
  }
}
```

**Screenshots**

![Image](https://github.com/user-attachments/assets/3277f1ee-989f-4d49-bf62-8ab1f4acec1f)


I'm dealing with this problem as well, it's a big problem when running large evaluations, since I like to check the progress while the eval is running.
Repository: promptfoo/promptfoo
Issue: Fix Request: "Show Failure only" adjustement
Root cause: Algorithmic Error

Hello ! I know it wont the game changing feature, but more Quality of Life feature

Supose you have two prompt an you tick " Show Falure only"

| Prompt 1 | Prompt 2 |
|----------|----------|
|   ✅     |    ❌    |
|   ✅     |    ❌    |
|   ❌     |    ✅    |
|   ✅     |    ❌    |
|   ❌     |    ✅    |


- Then hide the column Prompt 1 
In the current state we have  :

| Prompt 2 |
|----------|
|    ❌    |
|    ❌    |
|    ✅    |
|    ❌    |
|    ✅    |

Instead of only  this : 
| Prompt 2 |
|----------|
|    ❌    |
|    ❌    |
|    ❌    |

thanks 




Thanks for pointing this out!  This is fixed in af949db1f2a5125596a30816f97f0fcb29bc5ea7.  If a column is hidden, its failures won't be counted when deciding whether to display the row.
Repository: promptfoo/promptfoo
Issue: When trying to import a tool from a py file in the provider i get an error.
Root cause: Validation Gap

Describe the bug
When trying to import a tool from a py file in the provider i get an error.

To Reproduce

Conf 1 :

...
tools:
   - file://./tool.py

Conf 2 :

...
tools: [
   tool_1,
   tool_2
]
Repository: promptfoo/promptfoo
Issue: Https concatenation issue
Root cause: Validation Gap

Hi again, i just wanted to share that i had issues to call Azure OpenAI model cause i was filling the yaml like this :
```yaml
providers:
  - id: azureopenai:chat:gpt-35-deployement-name
    config:
      apiHost: https://apiAdress.azure.com
```

but  on execute it prefill the apiHost with an **https://**, giving 

>https://https://apiAdress.azure.com

instead of : 

>https://apiAdress.azure.com

Is it possible to implement a basic test (constains "https"...), to prevent any new user wasting time to detect this double https concatenation.

I know this is a dumb trap, i just need to readcarefully but when we copy URL from azure there is already the https. 

No one would thing to remove this https when copy pasting 😄 

Thanks

Is there any misunderstanding, or did I explain something incorrectly?

All clear, Theo!  Some other providers support `apiBaseUrl`, which behaves how you expected it to.  Perhaps it would make sense to add this config property to the Azure provider as well.

Yeah i totally agree, wouldn't it be interesting to add a little warning in the documentation maybe ?
Repository: promptfoo/promptfoo
Issue: Documentation is missing.


What are you trying to do that isn't documented?  https://promptfoo.dev/docs/intro

![image](https://github.com/promptfoo/promptfoo/assets/9823189/b613fcde-1f45-4980-9daa-2935e63efc1a)

I think the antivirus blocked the page, I tested it out in another PC and it worked, go figure.

I wanted to read the Guide and references, I thought the website was down, but It was my connection.

Ah ok. No problem!
Repository: promptfoo/promptfoo
Issue: Documentation is missing
Root cause: Documentation Deficiency

Hello, I was looking at this project thru a YouTube video that showcased it https://www.youtube.com/watch?v=KhINc5XwhKs&t=878s
I was about to test it out and looking for some more info I found the website down.

Could there be an issue with the domain https://promptfoo.dev/docs/intro ?

Thank you.
Excellent work by the way.


I think it might just be you?  The URL https://promptfoo.dev/docs/intro loads ok for me and on https://downforeveryoneorjustme.com/promptfoo.dev?proto=https

Resolved in #441
Repository: promptfoo/promptfoo
Issue: Feature Request: Gemini api (generative language api) support
Root cause: Configuration Error

Google offers the [Gemini API as a standalone API ](https://ai.google.dev/docs/gemini_api_overview)that only requires an API key, instead of complex authentication as needed in Gemini via Vertex AI. Anyone who does not have a Google Cloud account can easily access this API without using a credit card.

Although both APIs apparently offer the same model, in my experiments with models, the results differ. Therefore, I would like to test the standalone Gemini API using PromptFoo. This feature will hep developers use this version of gemini seamlessly.

Hi @mehnoorsiddiqui,

Thanks for pointing this out.  #445 updates the existing provider for Google's standalone API with Gemini support.  It will be in the next release published tonight.  You can use it like so:

```yaml
prompts:
  - "Write a short tweet about {{topic}}"

providers:
  - google:gemini-pro

tests:
  - vars:
      topic: bananas
```

(and set the `GOOGLE_API_KEY` environment variable)

This is part of v0.39.1

Hi, @typpo thanks for the release. But I am facing the API key error though I set in the .env file 
![image](https://github.com/promptfoo/promptfoo/assets/57288492/b29af16f-2072-4e51-85ec-6d1d89d8293b)


promptfoo doesn't load an .env file - you will have to actually set the variable in your environment

Thanks, @typpo. Now, I am not getting any authentication errors. However, I am unable to set the config as I was doing for the Gemini vertex API. Promptfoo is working without config but with config, facing the following error:
![image](https://github.com/promptfoo/promptfoo/assets/57288492/13ac6a43-ce70-4afc-bdd3-a5e12dd0d577)

I am setting the config like below:

```
providers:
  - id: google:gemini-pro   
    config:
      temperature: 0
      topP: 1
      
```
Repository: promptfoo/promptfoo
Issue: add AI/ML API as provider
Root cause: Unimplemented Feature Gap

Hi!

I'm Sergey from the Integrations team over at [AI/ML API](https://aimlapi.com/), a startup with 150K+ users, providing over 300 AI models in one place

Your project looks dope, so we'd like a native integration. We already got integrations with Langflow, AutoGPT, and LiteLLM - so our integrations team is pretty seasoned :)

Say you're interested, and we'll test the compatibility, update the code/docs to include us, create a PR and add a tutorial on using promptfoo with AI/ML API to our docs

If you guys give us a green light, we could get on it 👷

Hi @OctavianTheI! We would absolutely love an integration. You can see our docs for how to integrate a new llm provider here: https://www.promptfoo.dev/docs/contributing/#adding-a-new-provider Please feel free to reach out if you get stuck or need help! 

Thank you for contributing this! Closing as completed!
Repository: promptfoo/promptfoo
Issue: VertexAI models to grade metrics returns error 400
Root cause: Interface Contract Mismatch

Hi,

I've tested the default prompt config file and it works well with Vertex AI models as providers but when I use them as evaluators for testing like here: https://promptfoo.dev/docs/configuration/expected-outputs/model-graded#overriding-the-llm-grader
then I am getting the following error:

```
| [FAIL] Error 400: 1 instance(s) is allowed per prediction. Actual: 2 |
```
I've tried to override the LLM grader in different ways and got the same error. 

`promptfooconfig.yaml`:
```
# Learn more: https://promptfoo.dev/docs/configuration/guide
description: 'my description'

prompts:
  - "Write a tweet about {{topic}}"
  - "Write a very concise, funny tweet about {{topic}}"

providers:
  - vertex:gemini-pro
  - vertex:chat-bison

defaultTest:
  options:
    provider: vertex:gemini-pro

tests:
  - vars:
      topic: bananas

  - vars:
      topic: avocado toast
    assert:
      # For more information on assertions, see https://promptfoo.dev/docs/configuration/expected-outputs
      - type: icontains
        value: avocado
      - type: javascript
        value: 1 / (output.length + 1)  # prefer shorter outputs

  - vars:
      topic: new york city
    assert:
      # For more information on model-graded evals, see https://promptfoo.dev/docs/configuration/expected-outputs/model-graded
      - type: llm-rubric
        value: ensure that the output is funny
```

When I run `promptfoo eval`:
![image](https://github.com/promptfoo/promptfoo/assets/64668750/369cb65a-2c08-41f0-9f3a-c01cae53390d)

thanks for your help

The `llm-rubric` prompt (and other model-graded eval prompts) are written in OpenAI chat format.  This is usually ok, even for alternative models, because most APIs mimic OpenAI's format.  But Vertex is an exception!  PR #450 should solve this issue for Vertex.

Thank you for catching this issue!

thank you for your help! very much appreciated
I'd love to help on this great project but I need to improve on node.js first ahah 

This fix is published in 0.40.0
Repository: promptfoo/promptfoo
Issue: defaultTest to override embedding provider is not working
Root cause: Validation Gap

**Describe the bug**
When adding options to provide an embedding provider to override, an error is displayed. 

Error: Test case must contain one of the following properties: assert, vars, options, metadata, provider, providerOutput, threshold.

Instead got:
```json
{
  "defaultTest": {
    "options": {
      "provider": {
        "embedding": {
          "id": "bedrock:embeddings:amazon.titan-embed-text-v2:0",
          "config": {
            "profile": "ptx-master",
            "region": "us-west-2"
          }
        }
      }
    }
  }
}
```

**To Reproduce**
Add the following to config:

```yaml
defaultTest:
  options:
    provider:
      embedding:
        id: bedrock:embeddings:amazon.titan-embed-text-v2:0
        config:
          region: us-east-1
```



**Expected behavior**
The default embedding provider for tests using embeddings should be overridden


- Promptfoo version: 0.114.7



Can you share your full config?


defaultTest:
  options:
    provider:
      embedding:
        id: bedrock:embeddings:amazon.titan-embed-text-v2:0
        config:
          region: us-east-1description: 'the prompt generated by the script, ****PROMPT_DESCRIPTION****'
env: {}
prompts:
  - file://original-prompt.yaml
  - file://prompt-update-v1.yaml
providers:
  - id: 'file://../../promptfooProviders/qic_promptfoo_provider.py'
    label: 'qic provider'
    config:
      profile: 'ptx-master'
      region: 'us-west-2'
      assistantId: '067c9e25-7189-4fa5-ad36-3b9e240c910f'


scenarios: []
tests: file://tests.yaml
writeLatestResults: true
sharing: true

Hi @wobrien-pronetx,

Thanks for reporting this issue! You encountered a bug that was preventing `defaultTest.options.provider.embedding` from working correctly.

This was **fixed in PR #4732** (merged July 26, 2025) which specifically addressed your report. The fix allows `defaultTest` to contain partial test case properties like embedding provider overrides.

**Additional improvements:**
- PR #5348 (merged Aug 25, 2025) - Further improved `defaultTest.provider` handling
- PR #5931 (merged Oct 15, 2025) - Added full support for `defaultTest.options.provider` with model-graded assertions

**Your config should now work correctly:**
```yaml
defaultTest:
  options:
    provider:
      embedding:
        id: bedrock:embeddings:amazon.titan-embed-text-v2:0
        config:
          region: us-east-1

providers:
  - id: file://../../promptfooProviders/qic_promptfoo_provider.py
    label: qic provider

tests: file://tests.yaml
```

Please update to the latest version:
```bash
npm install -g promptfoo@latest
```

See the [Google provider docs](https://www.promptfoo.dev/docs/providers/google#overriding-providers) for more examples of provider overrides.

If you're still experiencing issues after updating, feel free to comment or re-open.

Thanks!
Repository: promptfoo/promptfoo
Issue: Add support for Mistral.ai
Root cause: Unimplemented Feature Gap

I saw that it's possible to use OpenRouter to run the mistral-medium model with promptfoo.

I have signed up for Mistral's `La Plateforme` and was making a custom provider work locally yesterday.

Is there any interest in adding native support?

If so, I'm happy to take a stab at it.

I definitely support adding Mistral!

If you've written the custom provider, you've done almost all the work already.  You'd have to add the `mistral:` prefix to [`providers.ts`](https://github.com/promptfoo/promptfoo/blob/main/src/providers.ts#L269-L271).  If you want to support caching, you can use [`fetchWithCache`](https://github.com/promptfoo/promptfoo/blob/main/src/cache.ts#L48) in your provider instead of regular `fetch`.

Let me know if you run into any issues - I'm also happy to help with this.

Ok, I'll clean up the custom provider a bit and create a PR.
Repository: promptfoo/promptfoo
Issue: Invalid JSON when using Anthropic API
Root cause: Validation Gap

**Describe the bug**
When trying to run custom prompt which does not have any special characters using multiple plugins, some POST requests succeed, while some fail stating
```
[HTTP Provider]: Response: "{\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"The request body is not valid JSON: unexpected character: line 8 column 316 (char 466)\"}}"
Provider response properties: output, tokenUsage
Provider response cached property explicitly: undefined
Evaluator response = {"output":{"type":"error","error":{"type":"invalid_request_error","message":"The request body is not...
Evaluator checking cached flag: response.cached = false, provider.delay = 0
Tracked token usage for http (HttpProvider): total=100, cached=0
[RedteamProviderManager] Loading redteam provider: {"providedConfig":"none","jsonOnly":true,"preferSmallModel":false}
```

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Create Red Team
2. Click on "I'm testing a model"
3. Select HTTP/HTTPS Endpoint
4. Select`Use Raw HTTP Request` and `Use HTTPS`
5. Add POST request in the body
6. In Plugin Configuration, click `Custom Prompts`. Enter a prompt
7. In Strategy Configuration, click `Custom` and unselect any other option having checkbox ticked.
8. Run

**Expected behavior**
All POST requests should get a valid response

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version: 0.115.0

**Additional context**
None.


Thanks for reporting this @rafsid!  The issue is with JSON serialization when using raw HTTP requests with complex data structures in red team testing. The quickest fix is to use promptfoo's native Anthropic provider instead:

```yaml
providers:
  - anthropic:messages:claude-sonnet-4-5-20250929
```

This will work seamlessly with red team testing and avoid the serialization issues entirely. If you still run into problems, feel free to reopen or create a new issue. Happy to help!
Repository: promptfoo/promptfoo
Issue: Feature Request: Use Promptfoo to generate DPO datasets
Root cause: Unimplemented Feature Gap

I use promptfoo a lot. It was the missing piece of a proper development stack for LLM-based applications.

As part of creating extensive evals, you get to compare a lot of different models with ease. That helps a lot with selecting the right model for a given task.

However, there will always be a goal to replace more expensive models with fine-tuned open-source alternatives.

Wouldn't it be great to take the existing data generated by running promptfoo to automatically create DPO datasets?

This could be as simple as using the UI to thumbs up one of the model responses per prompt.
Then all the ingredients should be pretty much there. The input system message/prompt, as well as a response pair with one response being marked as preferred.

Does that make sense?

This is an interesting idea, and you're right that we essentially have all the information already.  What is the simplest implementation that would be useful to you?  e.g. an "export" button or command that outputs {prompt, response, preferred} in a machine-readable format?

Glad to hear it!

Yes, an "export" button that produces a file with this format.

```json
[
    {
        "system": "You are an AI assistant...",
        "question": "Generate an approximately...",
        "chosen": "Midsummer House is a moderately...",
        "rejected": " Sure! Here's a sentence that..."
    },
    ...
]
```

#### User experience considerations

##### Evals for 2 models
- Allow setting a thumbs up for an entire column (e.g. auto-choose GPT-4 vs GPT-3.5)
- Allow filtering/exporting for only failed test cases, where it would automatically select the passing one as chosen and the failing one as rejected.





Hey @streichsbaer,

#482 starts the work on this.  A couple areas where I'm seeking feedback
- I've made `chosen` and `rejected` string arrays rather than strings, because some users have multiple columns.  Does that make sense to you, or does it make ingesting this data more cumbersome?
- I am assuming that `system` and `question` are test case vars.  Does that match workflow?  I'm wondering if there is a better way to do this more generically.  I could just add the `vars` map to the object.

Hi @typpo, thanks for that, and sorry for the late response.

- Strings should be fine for `chosen` and `rejected`, but having string arrays is ok too. The post-processing of this is not an issue.
- Correct, the `system` and `question` are the test case vars. Adding the vars map to the object is a good idea, then there is more flexibility in the post-processing steps.



This feature is released in [0.50.0](https://github.com/promptfoo/promptfoo/releases/tag/0.50.0).  Thanks and let me know if you have any more feedback!

![image](https://github.com/promptfoo/promptfoo/assets/310310/fd13001f-b9ff-4c43-86d3-512ca33c3441)


Awesome, thanks @typpo 🙏