zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Add metadata to CSV testcases
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**  
I have a test suite with 50 tests in my CSV file. I want to split them by topic manually and run only some of them.

**Describe the solution you'd like**  
Add a `__metadata:topic` column that would result in the value being put into the test's metadata, by which I can filter via `--filter-metadata`.

Note that some tests might cover 2 or more topics. In this case, I'd like to be able to put 2 values to the same metadata key as an array. For this, I propose a special column name convention: `__metadata:*[]`, e.g. `__metadata:topic[]`. In this case, the value should be split by `","` (with the ability to escape like this: `"\,"`).

**Describe alternatives you've considered**  
I'll probably use `--filter-pattern` as a workaround and put tags into the description. But this is brittle for larger test suites.

**Additional context**  
Even being able to do smth like `__metadata:topic` would be super-helpful. Being able to pass arrays is a cherry on top. 

In progress at #2709 

Hi @sbichenko,

Sorry for not closing this sooner - this has been working since **PR #2709** was merged back in March 2025 (version 0.107.5).

You can add metadata columns to your CSV files using the `__metadata:` prefix:

```csv
question,answer,__metadata:topic,__metadata:difficulty
What is 2+2?,4,math,easy
What is the capital of France?,Paris,geography,medium
```

For array values, use the `[]` suffix:
```csv
__metadata:tags[]
"tag1,tag2,tag3"
```

Then filter tests using:
```bash
promptfoo eval --filter-metadata topic=math
```

See the [Datasets documentation](https://www.promptfoo.dev/docs/configuration/datasets) for more details.

Thanks for the suggestion!
Repository: promptfoo/promptfoo
Issue: Mistake in documentation on JS in csv

Docs on [loading csv](https://www.promptfoo.dev/docs/configuration/expected-outputs#load-an-external-tests-file) mentions how to add JS assertions:

```
text,__expected
...
"Goodbye, everyone!","fn:return output.includes('Au revoir');"
```

This, however, crashes with javascript error. Using `"fn:output.includes('Au revoir');"` (omitting `return`) works fine.

I suggest fixing the docs.

Done in https://github.com/typpo/promptfoo-docs/commit/e3fa5af2cd00c14e6b8b0d7ba5d89df209c29922, thanks for catching!
Repository: promptfoo/promptfoo
Issue: Getting Started -  TypeError: pathOrGlob.startsWith is not a function
Root cause: Validation Gap

I am getting started by following the documentation 
https://www.promptfoo.dev/docs/getting-started

When I run the command
```
npx promptfoo@latest eval
```

I see the below error
```
TypeError: pathOrGlob.startsWith is not a function
    at /Users/[redacted]/Documents/my-learning/promptfoo/node_modules/promptfoo/dist/src/prompts.js:107:28
    at Array.flatMap (<anonymous>)
    at readPrompts (/Users/[redacted]/Documents/my-learning/promptfoo/node_modules/promptfoo/dist/src/prompts.js:106:45)
    at Command.<anonymous> (/Users/[redacted]/Documents/my-learning/promptfoo/node_modules/promptfoo/dist/src/main.js:258:57)
    at Command.listener [as _actionHandler] (/Users/[redacted]/Documents/my-learning/promptfoo/node_modules/commander/lib/command.js:482:17)
    at /Users/[redacted]/Documents/my-learning/promptfoo/node_modules/commander/lib/command.js:1283:65
    at Command._chainOrCall (/Users/[redacted]/Documents/my-learning/promptfoo/node_modules/commander/lib/command.js:1177:12)
    at Command._parseCommand (/Users/[redacted]/Documents/my-learning/promptfoo/node_modules/commander/lib/command.js:1283:27)
    at /Users/[redacted]/Documents/my-learning/promptfoo/node_modules/commander/lib/command.js:1081:27
    at Command._chainOrCall (/Users/[redacted]/Documents/my-learning/promptfoo/node_modules/commander/lib/command.js:1177:12)
```

Version Details
---
node version : v18.15.0
npm version:  9.5.0

Hi @gautam-goudar, try quoting the prompts:

```yaml
prompts:
  - 'Convert this English to {{language}}: {{input}}'
  - 'Translate to {{language}}: {{input}}'
```

Thank you.  Quoting the prompts worked. 

Thanks, documentation is now corrected
Repository: promptfoo/promptfoo
Issue: Google AI Studio Docs needs updating
Root cause: Documentation Deficiency

**Describe the bug**
The Google AI Studio provider docs are out of date

**To Reproduce**
Go to the Google AI Studio docs and review the supported models: https://www.promptfoo.dev/docs/providers/palm/

**Expected behavior**
The docs should reflect the latest models (1.5 and 2.0 are the only currently supported Gemini models)

**Screenshots**
<img width="1044" alt="Image" src="https://github.com/user-attachments/assets/a10587c6-d024-477d-a241-3a50789a100c" />

As an aside, it would also be helpful to cover how to make prompts and tests multi-modal. Gemini accepts images, audio, and video (as well as text). 

I'm having trouble adding images using the Python prompt provider for Gemini. Do you have any suggestions?

This has been updated!  thanks for flagging
Repository: promptfoo/promptfoo
Issue: Promptfoo eval with temperature 0 generates different responses for different users.
Root cause: Configuration Error

This is quite a big issue, because is hampering trust in the tool.

 I have a number of promptfoo tests in a github repo. The same code checked out by different users and executed with promptfoo eval generates different failing test cases for different users. 

Something else that I have noticed is that the exact same prompt generates slightly different responses from the actual app I am trying test cases for. I believe I am using the same prompts and settings (temperature of 0 and seed) and yet the app every now and then is giving slightly different responses from propmptfoo tests.

```
providers:
  - id: "openai:gpt-4o"
    config:
      temperature: 0.0
      seed: 111
``` 

Any idea on how i can fix the issue or I can check what's going on?

Hi Dario, can you run the eval with the --verbose flag? That way you'll be
able to see the actual payload sent to OpenAI in order to verify prompt,
params, etc are sent as intended.

On Fri, Jan 17, 2025, 5:02 AM Dario Bernardo ***@***.***>
wrote:

> This is quite a big issue, because is hampering trust in the tool.
>
> I have a number of promptfoo tests in a github repo. The same code checked
> out by different users and executed with promptfoo eval generates different
> failing test cases for different users.
>
> Something else that I have noticed is that the exact same prompt generates
> slightly different responses from the actual app I am trying test cases
> for. I believe I am using the same prompts and settings (temperature of 0
> and seed) and yet the app every now and then is giving slightly different
> responses from propmptfoo tests.
>
> providers:
>   - id: "openai:gpt-4o"
>     config:
>       temperature: 0.0
>       seed: 111
>
> Any idea on how i can fix the issue or I can check what's going on?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/2729>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJTNJOF67VTNM2PSIND2LD5ODAVCNFSM6AAAAABVL4B2MOVHI2DSMVQWIX3LMV43ASLTON2WKOZSG44TKNBTGE3TANA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
Repository: promptfoo/promptfoo
Issue: [Web UI] Unexpected Application Error! Minified React error #31
Root cause: Validation Gap

**Describe the bug**
Promptfoo web viewer does not start. 

**To Reproduce**

1. Run `promptfoo view`
2. Web browser shows:

```
Unexpected Application Error!
Minified React error #31; visit https://reactjs.org/docs/error-decoder.html?invariant=31&args[]=object%20with%20keys%20%7Bmethod%2C%20path%2C%20params%7D for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
Error: Minified React error #31; visit https://reactjs.org/docs/error-decoder.html?invariant=31&args[]=object%20with%20keys%20%7Bmethod%2C%20path%2C%20params%7D for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
    at kx (http://localhost:15500/assets/index-Di775SmD.js:38:6337)
    at f (http://localhost:15500/assets/index-Di775SmD.js:38:8126)
    at p (http://localhost:15500/assets/index-Di775SmD.js:38:9279)
    at b (http://localhost:15500/assets/index-Di775SmD.js:38:11357)
    at cs (http://localhost:15500/assets/index-Di775SmD.js:40:1479)
    at tX (http://localhost:15500/assets/index-Di775SmD.js:40:45814)
    at JY (http://localhost:15500/assets/index-Di775SmD.js:40:39822)
    at _Ae (http://localhost:15500/assets/index-Di775SmD.js:40:39748)
    at zS (http://localhost:15500/assets/index-Di775SmD.js:40:39598)
    at eP (http://localhost:15500/assets/index-Di775SmD.js:40:35960)
```
No error is shown in the terminal.

**Expected behavior**
Web viewer works.

**Screenshots**
![Image](https://github.com/user-attachments/assets/752d17ba-f48a-4d6f-95b9-a6cffb36c98c)

**System information:**

- Promptfoo version: `0.103.10`

**Additional context**
Latest version that works is `0.103.7`


Hi @rvernica thanks for reporting the issue! I fixed this in https://github.com/promptfoo/promptfoo/pull/2736 we'll get a release out today! 

Hi @rvernica, thanks again for reporting this issue! It has been resolved in version [0.103.11](https://github.com/promptfoo/promptfoo/releases/tag/0.103.11). Please update to the latest release, and let us know if you encounter any further issues.
Repository: promptfoo/promptfoo
Issue: Deepseek R1 is not working
Root cause: Configuration Error

**Describe the bug**
The model name of deepseek R1 is deepseek-reasoner https://api-docs.deepseek.com/quick_start/pricing . I'm trying to evaluate this model using promptfoo.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

consider the following config:
```yaml
providers:
### Deepseek
  - openai:chat:deepseek-chat:
      id: deepseek-chat
      config:
        apiKey: sk-
        apiBaseUrl: https://api.deepseek.com
  - openai:chat:deepseek-reasoner:
      id: deepseek-reasoner
      config:
        apiKey: sk-
        apiBaseUrl: https://api.deepseek.com
```
When invoke both deepseek models, we will see the following error:
```
API error: 400 Bad Request
{"error":{"message":"deepseek-reasoner does not support the parameter `presence_penalty`","type":"invalid_request_error","param":null,"code":"invalid_request_error"}}
```

**Expected behavior**
should be working just like deepseek-chat

**Screenshots**


**System information:**



- Promptfoo version:
0.103.8

**Additional context**



Hi @zhlmmc, thanks for the report.  https://github.com/promptfoo/promptfoo/pull/2753 should fix.  The root cause is that right now the provider is setting a `presence_penalty` by default.

> Hi [@zhlmmc](https://github.com/zhlmmc), thanks for the report. [#2753](https://github.com/promptfoo/promptfoo/pull/2753) should fix. The root cause is that right now the provider is setting a `presence_penalty` by default.

The latest release fixes this. Thanks!
Repository: promptfoo/promptfoo
Issue: OpenAI assistant support tool_resources config
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
To use OpenAI vector stores in assistants with their V2 beta, a config option `tool_resources` is required.

**Describe the solution you'd like**
Define the `tool_resources` config under the openAI provider and pass it into the assistant. 

**Describe alternatives you've considered**
None

**Additional context**
PR https://github.com/promptfoo/promptfoo/pull/2772 already created. I tested this locally for myself. 


Thanks for flagging this, it'll be in the next release :) 

Hi @dsappet, thanks for flagging this and for the #2772! tool_resources in OpenAI assistants is in the latest release ([0.103.13](https://github.com/promptfoo/promptfoo/releases/tag/0.103.13)). Please update to this version and let us know if you encounter any issues. Thanks for helping us improve Promptfoo!
Repository: promptfoo/promptfoo
Issue: How can I upload a file to be read during prompt testing with Promptfoo?
Root cause: Documentation Deficiency

 I am currently using Promptfoo and would like to test prompts that require uploading and processing a file (e.g., a PDF document or dataset). How can I configure Promptfoo to include a file upload as part of the testing process so that the file can be read and used during the evaluation?

Did you find a solution for the issue? Please post if you have any.

Hey @jaya437, you can load your file contents as a variable and then reference to it as base64:

https://www.promptfoo.dev/docs/configuration/parameters/#loading-images-and-other-file-types

Hi @funmise,

Thanks for your question about using Azure OpenAI for model-graded assertions.

You can specify Azure as the grader provider in your assertions:

```yaml
tests:
  - assert:
      - type: llm-rubric
        value: "Check if the output is helpful"
        provider:
          id: azure:chat:your-grader-deployment
          config:
            apiHost: your-azure-endpoint.openai.azure.com
            apiKey: ${AZURE_OPENAI_API_KEY}
```

Key points:
- Use `azure:chat:deployment-name` format
- Specify the grader provider in the assertion's `provider` field
- Include necessary Azure configuration (apiHost, apiKey, etc.)

See the [Azure Provider documentation](https://www.promptfoo.dev/docs/providers/azure) and [Model-Graded Assertions](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded) for more details.

Thanks!

Hi @funmise,

Thanks for your question about uploading files for testing!

As @nt9142 pointed out in the comments, you can load file contents as variables and reference them in your tests. This works for PDFs, datasets, images, and other file types.

See the documentation on [loading images and other file types](https://www.promptfoo.dev/docs/configuration/parameters/#loading-images-and-other-file-types) for details on how to configure this.

Hope this helps!
Repository: promptfoo/promptfoo
Issue: Go provider does not work with packages imported from internal folder
Root cause: Architectural Constraint

I have a issue relating to [GoProvider](https://www.promptfoo.dev/docs/providers/go/)

I have below folder structure:
```
--myproj
  --internal
    --pkg1
    --pkg2
    --evaluation
      --main.go
      --promptfooconfig.yml
```
In main.go, there is CallApi func.

In the main file, it imports files from pkg1 and pkg2. Something like:
```
package main

import (
  "github.com/myorg/myproj/internal/pkg1"
  "github.com/myorg/myproj/internal/pkg2"
)

...

func CallApi(...)
...
```

Now if I run `promptfoo eval` command under `evaluation` folder, I will see error log saying
```
2025-01-22T22:37:55.364Z [ERROR]: Error running Golang script: Command failed: go build -o /var/folders/yd/50q1qdp17fvcwwy4kvlt08p80000gn/T/golang-provider-kjc5Ax/golang_wrapper /var/folders/yd/50q1qdp17fvcwwy4kvlt08p80000gn/T/golang-provider-kjc5Ax/wrapper.go /var/folders/yd/50q1qdp17fvcwwy4kvlt08p80000gn/T/golang-provider-kjc5Ax/provider.go
package command-line-arguments
	/var/folders/yd/50q1qdp17fvcwwy4kvlt08p80000gn/T/golang-provider-kjc5Ax/provider.go:13:2: use of internal package github.com/myorg/myproj/internal/pkg1 not allowed

2025-01-22T22:37:55.365Z [ERROR]: Full error object: {"code":1,"killed":false,"signal":null,"cmd":"go build -o /var/folders/yd/50q1qdp17fvcwwy4kvlt08p80000gn/T/golang-provider-kjc5Ax/golang_wrapper /var/folders/yd/50q1qdp17fvcwwy4kvlt08p80000gn/T/golang-provider-kjc5Ax/wrapper.go /var/folders/yd/50q1qdp17fvcwwy4kvlt08p80000gn/T/golang-provider-kjc5Ax/provider.go","stdout":"","stderr":"package command-line-arguments\n\t/var/folders/yd/50q1qdp17fvcwwy4kvlt08p80000gn/T/golang-provider-kjc5Ax/provider.go:13:2: use of internal package github.com/myorg/myproj/internal/pkg1 not allowed\n"}
2025-01-22T22:37:55.369Z [ERROR]: Error running Golang script: Command failed: go build -o /var/folders/yd/50q1qdp17fvcwwy4kvlt08p80000gn/T/golang-provider-8Eks67/golang_wrapper /var/folders/yd/50q1qdp17fvcwwy4kvlt08p80000gn/T/golang-provider-8Eks67/wrapper.go /var/folders/yd/50q1qdp17fvcwwy4kvlt08p80000gn/T/golang-provider-8Eks67/provider.go
```

I don't know the internal implementation of go provider, but it seems like promptfoo is reading what I have in main.go and put it in another place, i.e. "main" package in go provider. And it will erroring out because Golang doesn't allow importing internal path from somewhere outside of internal path, even originally my evaluation folder is inside internal path.

My question being, has promptfoo go provider considered this scenario before and do you have any suggestions how should I be able to run `promptfoo eval` in this case?

Hi @zeqiancao, thanks for reporting this issue, we released a fix in [0.103.17](https://github.com/promptfoo/promptfoo/releases/tag/0.103.17). Please update and let us know if you encounter any further issues!
Repository: promptfoo/promptfoo
Issue: LLM models to test SQL queries
Root cause: Documentation Deficiency

How to connect with a backend api to fetch sql queries and provide expected queries and then test fetched queries vs expected queries and evaluating this using llm models

Hi @bharathiprodapt, check out https://www.promptfoo.dev/docs/guides/text-to-sql-evaluation/.  If you actually want to fetch live SQL results on-the-fly, you may have to write a custom provider.  For example: https://www.promptfoo.dev/docs/providers/python/
Repository: promptfoo/promptfoo
Issue: Tests running multiple times
Root cause: Algorithmic Error

When providing a defaultTest, it gets merged into test cases twice, resulting in duplicated test runs. 

When looking at your evaluator I see the offending code:

``` Javascript
 // Aggregate all vars across test cases
 let tests = (testSuite.tests && testSuite.tests.length > 0
            ? testSuite.tests
            : testSuite.scenarios
                ? []
                : [
                    {
                    // Dummy test for cases when we're only comparing raw prompts.
                    },
                ]).map((test) => {
            const finalTestCase = Object.assign({}, testSuite.defaultTest);
            return Object.assign(finalTestCase, test);
        });
```

Further down the defaultTest case is being added again:

``` Javascript
testCase.assert = [...(testSuite.defaultTest?.assert || []), ...(testCase.assert || [])];
```



Thanks for the report.  This should be fixed by https://github.com/promptfoo/promptfoo/pull/287
Repository: promptfoo/promptfoo
Issue: `contains` assert does not work with list from `js` file
Root cause: Validation Gap


**Describe the bug**
Following the [Load assertions from external file](https://www.promptfoo.dev/docs/configuration/expected-outputs/#load-assertions-from-external-file) documentation, I tried to use `not-icontains-any` assertion along with the blacklist of words. While the list of words from `.txt` file worked well, it didn't play well with javascript module.  

**To Reproduce**
`blacklist.js`:
```blacklist.js
module.exports = function () {
  return ['sorry'];
};
```

`promptfooconfig-bug.yaml`:
```promptfooconfig-bug.yaml
description: "bug demo"

prompts:
  - 'just say sorry about {{ topic }}'

providers: openai:gpt-4o

defaultTest:
  assert:
    - type: not-icontains-any
      value: file://blacklist.txt

tests:
  - vars:
      topic: 'the weather'
```

**Expected behavior**

The test should fail.


**Screenshots**

<img width="1183" alt="Image" src="https://github.com/user-attachments/assets/9bb952e9-6279-4c5e-aabf-d35ed8627c79" />

**Additional context**


I found that the the value returned by the js file is stored as `valueFromScript`: https://github.com/promptfoo/promptfoo/blob/b9092bc290fe3cd5268964d1012e9271c57d15ee/src/assertions/index.ts#L239

But the `contains` assertion does not take it into account:
https://github.com/promptfoo/promptfoo/blob/b9092bc290fe3cd5268964d1012e9271c57d15ee/src/assertions/contains.ts#L4-L9


Hi @mshavliuk, thanks for reporting this issue, we released a fix in [0.103.17](https://github.com/promptfoo/promptfoo/releases/tag/0.103.17). Please update and let us know if you encounter any further issues!
Repository: promptfoo/promptfoo
Issue: Feature request: ability to reference all outputs in tests
Root cause: Unimplemented Feature Gap

Unless I missed something in the documentation, in tests we can only access the output of one combination of provider/prompt. For each input data, I'd like to be able to compare the outputs of all prompts. For example, I'm testing prompts that revise manuscripts' abstracts. So if I have three prompts, I'd like to reference the three revised abstracts given one input. The idea of this is to ask another LLM model to grade them.

Based on my understanding of the documentation, in tests, we are limited to accessing the output from only one prompt at a time. However, for each input data, I would like to compare outputs from all the prompts. For instance, when testing prompts that revise manuscript abstracts, if I use three different prompts, I want to be able to review the three corresponding revised abstracts for a single input. So in a test, I'd like to be able to reference `output[0]`, `output[1]` and `output[2]`. The purpose of this is to have another LLM model evaluate and grade these revised abstracts together.

Hi @miltondp, thanks for the suggestion.  I understand the usefulness of being able to compare multiple outputs in a single prompt.  

You're correct that right now the eval runner executes each provider/prompt combo independently, rather than per row.  I'm interested in implementing this and will consider the best way to incorporate other outputs.

Hi @miltondp, this functionality has been added via the `select-best` assertion type in #438.  It will be in the next release.

For example:
```yaml
prompts:
  - "Write a tweet about {{topic}}"
  - "Write a very concise, funny tweet about {{topic}}"

providers: [openai:gpt-4]

tests:
  - vars:
      topic: bananas
    assert:
      - type: select-best
        value: choose the funniest tweet

  - vars:
      topic: nyc
    assert:
      - type: select-best
        value: choose the tweet that contains the most facts
```

Let me know what you think!

Very cool. I wonder if it is possible to use a prompt like [this one](https://chat.openai.com/share/be5a8b71-c423-4ffa-89b5-45d8855111f0). I guess that, internally, promptfoo does something similar.

I've added the ability to override this via the `rubricPrompt` test option.  Note that it expects the LLM response to contain the index of the winning output.  The available variables for the prompt are `criteria` (string) and `outputs` (list of strings).