zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: YAML File Test Case Upload via the Config Interface
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
In the Config Interface, it is only possible to upload a CSV version of the test cases

<img width="972" height="200" alt="Image" src="https://github.com/user-attachments/assets/257beb5b-e336-48ac-bf22-135e6ba9f6a3" />

**Describe the solution you'd like**
It would be great to be able to upload the YAML version of the test cases, currently supported when running the evals via the command line.



Hey @alexlssc, thanks for opening this.

YAML upload support for test cases was added in #5054 shortly after you filed this. The Config Interface now accepts `.yaml` and `.yml` files alongside CSV.

Both formats work:

```yaml
# Array of test cases
- description: Test 1
  vars:
    key: value

- description: Test 2
  vars:
    key: value2
```

```yaml
# Single test case
description: My test
vars:
  key: value
```

Closing this out, but let us know if you run into any issues with it.
Repository: promptfoo/promptfoo
Issue: Multi - step conversations with changing providers/toolcalls but keeping a comparison
Root cause: Unimplemented Feature Gap

I don't know if this is a feature request or just asking how to do something.

Situation
- Multi step conversational ai bot
- Each step can have different bots backing it and different tool calls provided 
  - So we thought using https://www.promptfoo.dev/docs/configuration/chat/#separating-chat-conversations to write tests
- But we need to be able to change the provider and toolcall at each test
- We also need to use promptfoo to compare prompts, previously this was dont by using different values in the top-level providers and a prompt.ts file that use the id to determine which to load.

So we want to be able to override the providers per test but still rerun them with different prompts, please advise and let me know if anything is unclear
Repository: promptfoo/promptfoo
Issue: Multiple prompts within one python file
Root cause: Documentation Deficiency

Hello, I really enjoy this library, and its helping my team tremendously.

I am wondering if there is any way to test out multiple prompts within one Python file. It would take too long to manually create the proper files and link them, and so we are using Python to format it properly, however we are unable to get 2 columns, where each column represents a different prompt.


```python
def generate_prompt(context: dict, X:list) -> str:
    return (
        f"""
        {X[0]}
        Describe {context['vars']['utterance']} concisely, create only one response from the AI.
---
        {X[1]}
        Describe {context['vars']['utterance']} concisely, create only one response from the AI.
        """
    )

if __name__ == "__main__":

    X = Y("T")

    x1, x2 = X.getx(["T.txt", "T_nof.txt"])

    print(
        generate_prompt(
            json.loads(
                sys.argv[1]
            ),
                [x1, x2],
        )   
    )
```

So as you can see within the code, we get 2 text files, and i want to compare them together. How can i evaluate on both of these prompts within one python script. 

Any help would be appreciated. Thank you :) 

There seems to be a way to do this [here](https://www.promptfoo.dev/docs/configuration/parameters) in Javascript as shown below, the question is how to do it within Python.
"
To reference a specific function in your prompt file, use the following syntax: filename.js:functionName:

```javascript
module.exports.prompt1 = async function ({ vars }) {
  return [
    {
      role: 'system',
      content: `You're an angry pirate. Be concise and stay in character.`,
    },
    {
      role: 'user',
      content: `Tell me about ${vars.topic}`,
    },
  ];
};
```
"

#533 adds the requested functionality.  It will be in the next release!

Released in 0.46.0
Repository: promptfoo/promptfoo
Issue: Broken nested JSON custom config in Red Team Targets pane
Root cause: Algorithmic Error

**Describe the bug**
When editing the "Custom Configuration" JSON element in Red Team evals "Targets", if you switch to another view (Usage Details, Plugins, etc.) and come back to the "Targets" section. You will have a modified JSON with nested "config" element.

Initial custom configuration:
``` json
{
  "temperature": 0.5
}
```

Final custom configuration:
``` json
{
  "temperature": 0.5,
  "config": {
    "temperature": 0.5
  }
}
```

if you repeat the modification step (one char is enough) and switch back the view, you will get another level of nesting with a new "config" key.


**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Run '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
No nested "config", I expect to just see my customization when I come back to the "Targets" view.

**Screenshots**

1. Customize base config:
<img width="812" height="523" alt="Image" src="https://github.com/user-attachments/assets/d03ce7f2-9dfb-432a-a61d-18ac81e218f3" />

2. Switch to another view and come back to "Targets" to see that:
<img width="812" height="523" alt="Image" src="https://github.com/user-attachments/assets/0537aa18-6345-4a47-a3f6-a34ed8497283" />


**System information:**

- Promptfoo version: **0.117.2**



Thanks for the bug report @gelim! I've resolved this issue in #5076, it should be included in the next release. 

This is now in the latest version!
Repository: promptfoo/promptfoo
Issue: Eval actions "Edit Name" not working
Root cause: Algorithmic Error

**Describe the bug**
When attempting to edit an eval using "Eval actions" / "Edit name" and filling out a new name, this is not reflected in the UI (after coming back to Evals / All Evals and refreshing).

**To Reproduce**

Do "Eval actions" and "Edit nam" any evals (running, finished). New name will not be taken into account.


**Expected behavior**
Take into account the new name for the eval.


**System information:**

- Promptfoo version: **0.117.2**
Repository: promptfoo/promptfoo
Issue: can we use repliacte's private models on promptfoo?
Root cause: Documentation Deficiency


Yes, this is supported.  Remember to set `REPLICATE_API_KEY` for your account with the private model.

Your config would look something like this:
```yaml
providers:
  - id: replicate:path/to/you/model:version123456
    config:
      temperature: 0.01
      # ...
```
Repository: promptfoo/promptfoo
Issue: ModelAudit - Critical Issues Not Reflected in Summary or UI Despite Being Present in Raw Output
Root cause: Algorithmic Error

Describe the bug
The dashboard summary shows 0 Critical issues, but the raw scanner output includes entries with "severity": "critical". The main screen also incorrectly shows a green “No error issues found” banner despite the critical-level content.

To Reproduce
Steps:

Run a model audit against a .pkl file containing posix or exec patterns.

Go to the “Results” tab.

Observe the summary showing “0 Critical” and a green success banner.

Scroll to the Raw Scanner Output — see critical-level issues listed.

Expected behavior
The top summary should reflect the correct number of critical issues (in this case, 2), and the banner should not say 0 critical errors, when critical items are present.

Screenshots

<img width="814" height="887" alt="Image" src="https://github.com/user-attachments/assets/2291a8a8-cb57-4742-91f2-fed462368ea3" />

Promptfoo version:
0.117.3


Is this the way it is intended to behave? 


Best Regards
Lukas
Repository: promptfoo/promptfoo
Issue: Could we get nunjucks and nunjucksFilters enabled for rubricPrompt?
Root cause: Unimplemented Feature Gap

I'm crafting a rubricPrompt but except variable expansion, no way to use any high level features of standard prompt, like nunjucks runtime and custom nunjucksFilters based on javascript.
By example, I would like to extract and post-process fields from a JSON output before analyzing them with a custom Model-graded metric.

Oops, looks like "options.transform" could help much. about output parsing.
But I need also access to original prompt."{{ prompt }}" doesn't work. Any advice?

That's a good suggestion - I've opened a PR that updates the `context` part of the transform function to include information about the prompt: https://github.com/promptfoo/promptfoo/pull/512

Good idea! I would be able to do all the stuff at transform level. Thx!

Works well, thank you @typpo !
Repository: promptfoo/promptfoo
Issue: Can not replicate consistants behavior of "promptfoo view" in self hosted app.
Root cause: Configuration Error

I tried to self host the app and noticed that with the default app memory option, the labels' modifications done in the ui did not remain, unless shared with the "Share" option which generates another link. 

Also, when an evaluation is sent to the local backend, it disappears when the app is stopped and run again, which is not the case when shutting down the server raised by the "promptfoo view" command.

I believed I had to provide memory to my app and tried to set the "PROMPTFOO_SHARE_STORE_TYPE" to "filesystem" and give a dedicated folder.

It does not seem to have changed much, even though once i run the docker image i get a log stating "Using Filesystem store" and in both cases (memory and filesystem storage),when I share an evaluation, I can see a line "Storing eval result with id f:caf6b..." written in my CLI. 

How can I add consistency to the evals I share to keep them when i shut down the app, and when they are modified in the UI so that I do not need to generate another link everytime ?

Filesystem storage type should solve this problem. Are you sure you are
mapping the docker volume to a persistent location on the host machine,
using the -v arg for docker run?

On Thu, Feb 29, 2024, 9:18 AM RayannBen ***@***.***> wrote:

> I tried to self host the app and noticed that with the default app memory
> option, the labels' modifications done in the ui did not remain, unless
> shared with the "Share" option which generates another link.
>
> Also, when an evaluation is sent to the local backend, it disappears when
> the app is stopped and run again, which is not the case when shutting down
> the server raised by the "promptfoo view" command.
>
> I believed I had to provide memory to my app and tried to set the
> "PROMPTFOO_SHARE_STORE_TYPE" to "filesystem" and give a dedicated folder.
>
> It does not seem to have changed much, even though once i run the docker
> image i get a log stating "Using Filesystem store" and in both cases
> (memory and filesystem storage),when I share an evaluation, I can see a
> line "Storing eval result with id f:caf6b..." written in my CLI.
>
> How can I add consistency to the evals I share to keep them when i shut
> down the app, and when they are modified in the UI so that I do not need to
> generate another link everytime ?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/510>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJQIS4IJ5QSWOMW2MTLYV5RFBAVCNFSM6AAAAABEAKXL4WVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE3DCNZUGQYTEMY>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
Repository: promptfoo/promptfoo
Issue: Gender Bias "view logs" built the wrong filter
Root cause: Algorithmic Error

**Describe the bug**

When clicking on "View logs" for "Gender Bias" vulnerabilities and mitigations from the Vulnerability Report page we don't see any logs.

**To Reproduce**

1. Access Vulnerability Report page in a red team eval
2. Scroll down to "Gender Bias" line and click on "View logs"
3. See the wrongly filter built "Metric: Gender Bias"


**Expected behavior**

Filter should be something of "Metric: BiasGender"
as I see there are sub categories, there should be a logical OR chained if wanting to get the full BiasGender/*
or support substrings operator like "contains"


**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version: **0.117.4**


Thanks for reporting, @gelim. This is fixed within #5086, and I will follow up here once it has been released.

The fix should be live for this. @gelim Let us know if you still encounter this issue, thanks!
Repository: promptfoo/promptfoo
Issue: How do you set a name for a provider when it's coming from a separate file?
Root cause: Unimplemented Feature Gap

Currently, I can write:
```yml
providers:
  - openai:gpt-3.5-turbo
    id: openai-gpt-3.5-turbo-hightemp
    config:
      temperature: 0.9
```
This means the model is `openai:gpt-3.5-turbo`, and the name of the model that's also visible in the output HTML is `openai-gpt-3.5-turbo-hightemp`.

If the provider is coming from a file like this one:
```yml
providers:
  - file://providers/gpt-3.5-turbo-lowtemp.yaml
```
Then the provider file looks like this:
```yml
id: openai:gpt-3.5-turbo
config:
  temperature: 0.2
```
Here, the ID refers to the model, but I couldn't find a way to specify the name. So, what ends up happening is that I don't see the names of the models in my HTML output anymore.

Thanks for raising this, you're right that it's an issue when you include the provider in the way that you described.  I'd like to add a `label` field or similar to the provider object, and get rid of the map that is used right now to label providers.  That will make it easy for you to add a label.

#563 adds this capability, e.g. with the following setup:

promptfooconfig.yaml:
```yaml
prompts:
  - 'Respond to the following instruction: {{message}}'

providers:
  - file://lowtemp.yaml
  - file://hightemp.yaml

tests:
  - vars:
      message: What's the capital of France?
  # ...
```
lowtemp.yaml:
```yaml
id: openai:gpt-3.5-turbo-0613
label: openai-gpt-3.5-turbo-lowtemp
config:
  temperature: 0
```
hightemp.yaml:
```yaml
id: openai:gpt-3.5-turbo-0613
label: openai-gpt-3.5-turbo-hightemp
config:
  temperature: 1
```

Released in [0.48.0](https://github.com/promptfoo/promptfoo/releases/tag/0.48.0)!
Repository: promptfoo/promptfoo
Issue: List reasoning tokens on the left side of the token breakdown tooltip
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
When I hover over the token information for a Gemini request with thinking/reasoning on, I get this:

<img width="360" height="106" alt="Image" src="https://github.com/user-attachments/assets/e3ef682f-4031-4af3-b85f-dc2922b2024b" />

This reads like an equation, but the numbers on the left side and the right side of the `=` aren't equal.

**Describe the solution you'd like**
Move the number of reasoning tokens to the left side of the `=`, so that it reads:

```
30,545 prompt tokens + 3,226 completion tokens & 4,115 reasoning tokens = 37,886 total tokens
```

This would be much less confusing.

Thanks for reporting this @sobjornstad! Fixed in #5113. Feel free to open another issue if you run into anything else!
Repository: promptfoo/promptfoo
Issue: Gemini 2.5 Pro via OpenRouter Only Outputs Thinking Process, Never Actual Response
Root cause: Interface Contract Mismatch

# Gemini 2.5 Pro via OpenRouter Only Outputs Thinking Process, Never Actual Response

Version: 0.117.4
## Description
Gemini 2.5 Pro through `openrouter:google/gemini-2.5-pro` outputs thinking tokens but never produces the requested response, even when explicitly stating it will format the output.

## Reproduction
```yaml
providers:
  - id: openrouter:google/gemini-2.5-pro
    config:
      temperature: 0.3
      max_tokens: 16000

prompts:
  - |
    [Any prompt requesting structured output]
    
    Format your response like this:
    <transcript>Your answer</transcript>
    <confidence>red/yellow/green</confidence>
```

## Result
Model output ends with:
```
**Formatting the Response**
I've finalized the transcript. The response will now be formatted as specified.
```
Then stops. No XML output despite:
- Using only 2,161 of 16,000 available tokens
- `finishReason: "stop"` (not a limit issue)
- Model explicitly saying it would format the response

## Expected vs Actual
- **Expected**: Thinking process followed by XML formatted response
- **Actual**: Only thinking process, stops before outputting any XML

## Control Test
Same prompt with `openrouter:anthropic/claude-3.5-sonnet` works perfectly, outputs proper XML.

## Analysis
This appears to be promptfoo not properly parsing Gemini thinking model responses, possibly because the thinking and actual response are in different parts/candidates of the response object.

## Impact
Makes Gemini 2.5 Pro unusable for structured output generation through promptfoo.
Repository: promptfoo/promptfoo
Issue: Persist changes on self-deployed UI without sharing a new link
Root cause: Unimplemented Feature Gap

Hello, I am trying to setup promptfoo for my organisation and hence using the self-deployed option.

I want to be able to alter the labels manually in the UI and persist the changes, without having to create a new link. Is it possible?

I am using the file-system store

#538 implements this.  Worth noting that this means _anyone_ with access can make changes.  User accounts/ACL/etc are future projects, but hope this helps in the meantime.

Released in 0.46.0
Repository: promptfoo/promptfoo
Issue: confused on csv asserts
Root cause: Documentation Deficiency

Greetings,

thanks for this amazing tool! I have the following format csv

```csv
source_sentence,translated_sentence,targetLanguage,__expected

"<span>The convention city of Glasgow has a rich history</span>, <b>fascinating culture</b>, and beautiful architecture.","<span>Stævnebyen Glasgow er rig på historie,</span> <b>fascinerende</b> kultur og smuk arkitektur.","Danish","contains-any: <\b> </span>"
```

this fails on the `contains-any` do I need to delimit those values? If so how?

Thanks!

Hi @yosiasz! Thank you so much.  The issue is that `contains-any` splits values by commas, not spaces. Also you have `<\b>` which should be `<b>`.

Change:
```csv
"contains-any: <\b> </span>"
```

To:
```csv
"contains-any: <b>,</span>"
```

Your current format is looking for the exact string `"<\b> </span>"` including the space, which won't match.

## Working example

```csv
source_sentence,translated_sentence,targetLanguage,__expected
"<span>The convention city of Glasgow has a rich history</span>, <b>fascinating culture</b>, and beautiful architecture.","<span>Stævnebyen Glasgow er rig på historie,</span> <b>fascinerende</b> kultur og smuk arkitektur.","Danish","contains-any: <b>,</span>"
```

With this config:
```yaml
prompts:
  - "{{translated_sentence}}"

providers:
  - id: echo

tests: file://your-tests.csv
```

This will pass since the output contains both `<b>` and `</span>` tags.  I hope this helps! Feel free to ask a follow up. I'll leave the issue open for you if you have any more questions. 

Awesome let me try that.

that does it thanks so much!

is this comma delimited example missing from the documentation?
Repository: promptfoo/promptfoo
Issue: contains-html
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
It would be nice to include a `contains-html` feature instead of specifying html tag that can a bit tedious

**Describe the solution you'd like**
Ability to use `contain-html` without having to specify html tags

**Describe alternatives you've considered**
adding all possible html tags in `contains-any`

**Additional context**
Gracias
Repository: promptfoo/promptfoo
Issue: Huggingface Api key through config results in error call
Root cause: Configuration Error

When using api key in configurations for an huggingface provider ([Tinyllama](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)), the call results in the following error:
`API call error: The following model_kwargs are not used by the model: ['apiKey'] (note: typos in the generate arguments will also show up in this list)`

I suppose that the api key should be removed from configurations when the package makes the api call and should be used only in headers.

Thanks for the catch!  #519 should fix this

Thank you for your wor 

Fix released in 0.44.0
Repository: promptfoo/promptfoo
Issue: Use a language‑specific tokenizer (e.g. jieba for Chinese)

Greetings

let us say I have this csv

```
source_sentence,translated_sentence,targetLanguage
Thank you very much,非常感谢,zh-Hans
```

Is there a way to use an external tokenizer to split/chunk the Chinese sentence

```
    assert:
      - type: javascript
        value: |
          // Split both strings into word arrays, normalize to lowercase
          const outputWords = output.split(/\s+/).map(w => w.toLowerCase());
          const expectedWords = vars.translated_sentence.split(/\s+/).map(w => w.toLowerCase());

          // icontains-any + whole words: pass if any expected word is in outputWords
          return expectedWords.some(word => outputWords.includes(word));

```

hope my question makes sense.

Thanks

Hi @yosiasz,

You can use external tokenizers! JavaScript assertions in promptfoo can run arbitrary code, including requiring npm packages. For Chinese tokenization, I recommend using nodejieba:

```
  npm install nodejieba
```

Then create an external file for your assertion:

assertions/chinese-tokenizer.js

```js 
const nodejieba = require('nodejieba');

module.exports = (output, context) => {
    const outputWords = nodejieba.cut(output);
    const expectedWords = nodejieba.cut(context.vars.translated_sentence);

    // Check if any expected word appears in the output
    return expectedWords.some(word => outputWords.includes(word));
};

```

In your config:

```yaml
  assert:
    - type: javascript
      value: file://assertions/chinese-tokenizer.js
```

  This way you can use proper Chinese word segmentation that correctly splits "非常感谢" into ["非常", "感谢"].

  Hope this helps!
Repository: promptfoo/promptfoo
Issue: assertion pass when it should'nt
Root cause: Validation Gap

As you can see in the screenshot, the answer should pass when the result contains 72, 
but gpt4 answers 73, which is not the expected value.
Maybe a mix between string and number but it's confusing.

![image](https://github.com/promptfoo/promptfoo/assets/1949589/9bf4cb43-2c0a-42e7-8384-7c7b23ddf28d)


damn yaml, I missed a tab and the assertion was seen as a variable...
sorry for the noise

yaml problems :(  it happens
Repository: promptfoo/promptfoo
Issue: Google provider fails with WebP images
Root cause: Validation Gap

When using a webp image with the Google Gemini provider, the request returns nonsense results.

This appears to be caused by the file format checks in `src/providers/google/util.ts`. They check that a WebP file should starts with the Base64-encoded string ("UklGRg"). It should be "UklGR". That's what the check in `src/providers/hyperbolic/image.ts` does.

The exact reason is that WebP files start with the ASCII characters "RIFF" and then the filesize as a uin32. That's 32-bits for "RIFF". But Base64 uses 6-bit groupings. That maps to "UklGR" (5 characters * 6 bits = 30 bits). So there are two bits left over, which carry over into the next base64 grouping. Since the next bytes in the webp file are the file size, which is variable, the next character is variable. So the final "g" should be removed from the file type check.

looks like this was fixed in a previous release!