zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Filter Targets
Root cause: Architectural Constraint

**Is your feature request related to a problem? Please describe.**

We want to run a series of test on dev/stage/prod environment. Right now we have `promptfoo.dev.yaml` etc and each has a `target` to the same provider but with different config (different username/password/url).

Alternatively, we should be able to have a single `promtpfoo.config.yaml` and have

```
targets:
  - id: aProvider
     label: dev
     config:
       username: user1
       password: pass1 
  - id: aProvider
     label: stage
     config:
       username: user2
       password: pass2
```

Then do `promptfoo eval --filter-target dev` 

If that makes sense, I can make a PR to add support for this. Thoughts/suggestions/alternatives? 


We actually support this through the `--filter-providers` flag on `eval`.  It takes a regex which matches on `id` and `label`.  Here's the implementation: https://github.com/promptfoo/promptfoo/blob/main/src/commands/eval/filterProviders.ts#L3

`targets` is just an alias for `providers`, which I know is confusing.  #1863 adds support for `--filter-targets`.  Hope this helps

@typpo I tried that, but that filters by the `id`. I would like to filter by `label` since it's the same provider with different configuration 

hmm, it _should_ match the label as well: https://github.com/promptfoo/promptfoo/blob/main/src/commands/eval/filterProviders.ts#L17

@typpo okay thanks ya that works. However, there is a potential bug. Here is our custom provider:

```
targets:
  - id: file://customProvider.js
    label: Dev
    config:
      env: 'dev'
      username: abc1
      passwordKey: PASS_KEY1

  - id: file://customProvider.js
    label: Stage
    config:
      env: 'stage'
      username: abc2
      passwordKey: PASS_KEY2
```

Inside the `customProvider` I am doing a check if `if (!process.env[options.config.passwordKey]) { /* error out */ })` 

What is happening now is even if I run `promptfoo eval --filter-providers Dev`, `promptfoo` is first loading the providers (by instantiating them) before filtering them. So I unless I provide both environment variables, I get an error and the app crashes. 

I am guessing we need to update `--filter-providers` to read a static `.name/.label` from the functions so it can do a filter pre-instantiation. 

Thughts? I can make a PR here

Hi @ktalebian, we'll look into adjusting the behavior so that providers are only instantiated after filtering. As a potential workaround, you could delay loading the custom provider library until the first LLM call inside callApi, ensuring the environment variables are only required once it’s actually used. If you'd like to make a PR for this, I'd be happy to review it! If not, I’ll try to find time to address it this week. Thanks again for the PRs and all your helpful suggestions—I really appreciate your contributions. Please feel free to send me your address, and I'll send you some Promptfoo swag as a thank you!
Repository: promptfoo/promptfoo
Issue: dry-run for cost estimation
Root cause: Unimplemented Feature Gap

Is there a feature that allows for a dry run of the evaluation prior to execution? Considering that evaluations incur costs related to models and tokens, the ability to estimate approximate pricing through a dry run would offer valuable guidance for budgeting. If a dry run feature is not available, could you recommend alternative methods for price estimation? For instance, would integrating Langfuse, which clarifies how model pricing is determined based on model type/version and tokens/characters, be a viable option?

Hi @hansujin93, thanks for the suggestion! I think it's a great idea, but it's quite challenging to implement an accurate dry run cost estimator due to the variability in tokenization across different models. Since it's difficult to predict how many tokens an LLM call will consume beforehand, we likely won't support this feature in Promptfoo.

Check out [LiteLLM's cost estimator](https://docs.litellm.ai/docs/completion/token_usage), which does a great job of calculating token usage after you've made an API call. It might be helpful for your needs. Thanks again for the suggestion. 

Please feel free to post any other questions or issues you come across.
Repository: promptfoo/promptfoo
Issue: `equals` failure description seems inverted
Root cause: Validation Gap

Hi, I started using PromptFoo recently and it's been really helpful for doing some evals! Wanted to raise something small. Open to being incorrect about the expected behavior since I am new to this library. Also, perhaps the behavior is correct but the documentation is misleading somehow.

**Describe the bug**

The [docs for `equality`](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#equality) say:

> The equals assertion checks if the LLM output is equal to the expected value.

> Example:

```
assert:
  - type: equals
    value: 'The expected output'
```

However, when I specify the following `test` as one of the `tests`: 

```
{
  "type": "equals",
  "value": "0",
  "transform": "output.answer",
},
```

If it fails (perhaps the model returns `{"answer": "1"}`), I see an failure message of:

`[FAIL] Expected output "0" to equal "1"`

I would expect it to be the other way around:

`[FAIL] Expected output "1" to equal "0"`

Since the output was actually 1.

**To Reproduce**

Here is a small JSON config test case:

```
{
  "providers": ["echo"],
  "prompts": ["1"],
  "tests": [
    {
      "assert": [
        {
          "type": "equals",
          "value": 0
        }
      ]
    }
  ]
}
```

=>

```
┌──────────────────────────────────────────
│ [echo] 1
├──────────────────────────────────────────
│ [FAIL] Expected output "0" to equal "1"
│ ---
│ 1
└──────────────────────────────────────────
```

**Expected behavior**

See above.

**Screenshots**

N/A

**System information (please complete the following information):**

- Promptfoo version: 0.92.2 (latest)

**Additional context**

N/A

Hi @panozzaj, thanks for finding this! I completely agree. Here is the relevant line: https://github.com/promptfoo/promptfoo/blob/main/src/assertions.ts#L389. I’d be happy to accept a PR from you if you'd like credit for fixing it. If not, I’ll take care of this later today.

OK, I will do that now and PR, thanks for the pointer!
Repository: promptfoo/promptfoo
Issue: Make the eval result table scrollable
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
When there are a large number of results for multiple prompts, scrolling down past the header makes it difficult to tell which prompt a result belongs to.

When there are a number of different prompts being evaluated (7+) the resulting vars and results are squished.

**Describe the solution you'd like**
Don't specify fixed widths.

Make the top header sticky (`position: sticky`) to fix it when scrolling down the page.

Make the left header sticky to fix it when scrolling down the page.

**Describe alternatives you've considered**
N/A

**Additional context**
N/A


Done in latest version!
Repository: promptfoo/promptfoo
Issue: Allow running the same test multiple times in a single run, and cache all results together
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
AI is unstable, sometimes the same test run will work and then fail, making it hard to determine whether a test failed or not

**Describe the solution you'd like**
An optional config parameter that defines  the number of times to run each test. The report should show a summary of the results and the cache should cache all of the 

**Describe alternatives you've considered**
Alternative would be to have several separate cache-store, and rerun the test against each one of these and somehow combine the results 


Hi @QuarkNerd, thanks so much for your suggestion! Promptfoo already has a feature that might help with what you're looking for: the `--repeat` command. It doesn’t cache results at the moment, but you can create your own cache with a custom provider or prompt function if that fits your use case. We do have caching with `--repeat` on our roadmap. 

Here’s how you can use the command:

```bash
promptfoo eval --repeat 5
```

If you have any other specific needs or ideas for improvement, we’d love to hear them. 

Thanks @mldangelo I will check those out.

Do you have an approx timeline for this? (I won't hold to anything, but looking at approximations). And do you envisioning it being overly complicated? I would be willing to take a look at it myself, my inital thoughts would be to pass repeatIndex down to `fetchWithCache` or whatever uses `getCache` by updating the providers accordingly. Maybe in a cacheOptions object to avoid ballooning more parameters.
The index could then be added to the cacheKey with no change for repeatIndex = 0.

This is based on my own judgment of the code and it's quite possible I'm overlooking somethings, please let me know your thoughts
Repository: promptfoo/promptfoo
Issue: How to run factuality tests on a CSV file?
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

I'm trying to run factuality tests on a CSV file, where I have two columns: `question` and `__expected`. The goal is to send the `question` to an assistant API and compare the response with the `__expected` output, which contains the ideal response.

However, I'm facing issues when trying to define metrics for the entire file. I tried providing `- file:` and `- assert:`, but it fails. While defining metrics per row is possible, it's not convenient, and the factuality tests still fail when the assistant's response is a superset of the expected result.

**Describe the solution you'd like**

I would like a way to define metrics for the entire file and allow for flexible comparisons where the assistant's response can include more details than the `__expected` output but still pass the test if the factual content is correct. This would help when the assistant provides additional context that doesn't invalidate the factuality of the response.

**Describe alternatives you've considered**

I have considered defining metrics per row, but this approach doesn't seem to work as expected for factuality tests. Even when the assistant's response is a superset of the expected result, the test fails. Ideally, I'd prefer a simpler way to handle this at the file level.

**Additional context**

Example YAML configuration I'm using:

```yaml
description: "Factuality evaluation of Assistant"

prompts:
  - "{{question}}"

providers:
  - openai:assistant:asst_id-XYZ

defaultTest:
  options:
    provider: openai:gpt-4o-mini

tests: file://short-testset.csv
```

Example CSV that fails:

```csv
question,__expected,__metric
what GPT stands for?,"GPT stands for ""Generative Pre-training Transformer""",factuality
```

Ideal CSV:
```csv
question,__expected
what GPT stands for?,"GPT stands for ""Generative Pre-training Transformer"""
```

Ideal YAML:
```yaml
tests:
  - file: file://short-testset.csv
    assert:
      - type: factuality
```
I'm looking for guidance on how to define the metrics and handle the superset comparison for factuality testing. Thanks in advance 


So, I ended up writing a custom script that transforms CSV files into YAML while adding the same 'factuality' check for every use case. Additionally, my script checks if a question already exists and adds only new ones.

I also discovered that in the PromptFoo UI under the creation section, you can upload a CSV. However, it doesn't support defining your desired assert conditions and it just set the equals assertion uniformly for all tests. Moreover, it lacks an update option.

If anyone is interested, I can share the script. Feel free to reach out.
Repository: promptfoo/promptfoo
Issue: "You must provide at least 1 prompt"
Root cause: Documentation Deficiency

**Describe the bug**

I compile an evaluation successfully and try to run it again, but I always get the message “You must provide at least 1 prompt” .

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Run `promptfoo eval` (works well)
2. Run `promptfoo eval`
4. See error

- Promptfoo version: 0.93.1



Wondering if you have a standalone repro?  I am not seeing this on my end.

Hello @typpo, no. I'm using a regular folder on my C: 

Check the following image pls:
![image](https://github.com/user-attachments/assets/46313ab0-ee87-4eef-ad52-563e8edbb184)


@joelcarvalhopribss where are you running `promptfoo eval` relative to your file? Can you try to specify the path to the config with `promptfoo eval -c path/to/config.yaml` and see if that works? 

You are missing a `-c`:

```
promptfoo eval -c test.yaml
```
Repository: promptfoo/promptfoo
Issue: assertion values should support array of files

Instead of doing the following:
```
- description: 'Testing Full Summary prompt quality'
    vars:
      input: file://evaluation/inputs/full-1.txt
    assert:
      - type: contains-json
      - type: similar
        value: file://evaluation/outputs/full-1.txt
        threshold: 0.8

  - description: 'Testing Full Summary prompt quality'
    vars:
      input: file://evaluation/inputs/full-2.txt
    assert:
      - type: contains-json
      - type: similar
        value: file://evaluation/outputs/full-2.txt
        threshold: 0.8

  - description: 'Testing Full Summary prompt quality'
    vars:
      input: file://evaluation/inputs/full-3.txt
    assert:
      - type: contains-json
      - type: similar
        value: file://evaluation/outputs/full-3.txt
        threshold: 0.8

  - description: 'Testing Full Summary prompt quality'
    vars:
      input: file://evaluation/inputs/full-4.txt
    assert:
      - type: contains-json
      - type: similar
        value: file://evaluation/outputs/full-4.txt
        threshold: 0.8
```

**Describe the solution you'd like**
I would love to be able to do:
```
- description: 'Testing Full Summary prompt quality'
    vars:
      input:  
        - file://evaluation/inputs/full-1.txt
        - file://evaluation/inputs/full-2.txt
        - file://evaluation/inputs/full-3.txt
        - file://evaluation/inputs/full-4.txt
        
    assert:
      - type: contains-json
      - type: similar
        value: 
          - file://evaluation/outputs/full-1.txt
          - file://evaluation/outputs/full-2.txt
          - file://evaluation/outputs/full-3.txt
          - file://evaluation/outputs/full-4.txt
        threshold: 0.8
```


This was just released in [0.93.3](https://github.com/promptfoo/promptfoo/releases/tag/0.93.3) Thanks @danpe!
Repository: promptfoo/promptfoo
Issue: Vulnerability report for red teaming evaluation fails to show in the web app with: "Cannot read properties of undefined (reading 'type')"
Root cause: Validation Gap

**Describe the bug**
The vulnerability report for red teaming evaluation fails to show in the web app. When I click on the button to see the report, the page displays an error:

```
Unexpected Application Error!

Cannot read properties of undefined (reading 'type')

TypeError: Cannot read properties of undefined (reading 'type')
    at http://localhost:15500/assets/index-BPrn51S5.js:767:15637
    at Array.map (<anonymous>)
    at WZe (http://localhost:15500/assets/index-BPrn51S5.js:767:15605)
    at $P (http://localhost:15500/assets/index-BPrn51S5.js:38:17031)
    at zW (http://localhost:15500/assets/index-BPrn51S5.js:40:44104)
    at QW (http://localhost:15500/assets/index-BPrn51S5.js:40:39822)
    at Gae (http://localhost:15500/assets/index-BPrn51S5.js:40:39748)
    at J2 (http://localhost:15500/assets/index-BPrn51S5.js:40:39598)
    at V4 (http://localhost:15500/assets/index-BPrn51S5.js:40:35960)
    at UW (http://localhost:15500/assets/index-BPrn51S5.js:40:34907)
```

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Run a red teaming evaluation `npx promptfoo redteam run`
2. Run 'npx promptfoo view'
3. In the web browser, in the promptfoo web app, select `Evals`
4. Select a red teaming evaluation in the dropdown
5. Click on 'Vulnerability Report'
7. See error

**Expected behavior**
The vulnerability report must be displayed.

**Screenshots**
If applicable, add screenshots to help explain your problem.

![image](https://github.com/user-attachments/assets/fcaa5b42-6ed2-4d8b-b014-5535fc551e89)

**System information (please complete the following information):**

- Promptfoo version: 0.93.1

**Additional context**
N/A.


Hard to tell with the minified error, but I _think_ this was introduced in 0.93.0 with 04e45639a3e0f491e159b0ca7143473a273c3b90.  I'll get a fix in.  You may be able to temporarily work around by using version 0.92.3.

@typpo This now works correctly with v0.93.3, good work!

Thanks!

Great!
Repository: promptfoo/promptfoo
Issue: bug: config (yaml) output
Root cause: Documentation Deficiency

Defining
```yaml
output: output.json
```
does not generate a output file on 0.8.3.

This key changed to `outputPath` in the yaml config.  Great catch, I think this is not documented adequately.

If you aren't already on it, I can update the docs.

No worries.  All set here: https://www.promptfoo.dev/docs/configuration/guide#config

I think the cli 's `output` [argument](https://github.com/typpo/promptfoo/blob/cdbf80676e24bc7cdcc3f15a725f2d61c4529631/src/main.ts#LL136C1-L136C1) should then also be changed to `outputPath` for consistent naming.
Repository: promptfoo/promptfoo
Issue: Unable to open promptfoo link from `promptfoo share`
Root cause: Resource Mishandling

```
Uncaught (in promise) QuotaExceededError: Failed to execute 'setItem' on 'Storage': Setting the value of 'ResultsViewStorage' exceeded the quota.
    at Object.setItem (921-ea4bddb5c25058ea.js:44:2429)
    at v (921-ea4bddb5c25058ea.js:44:4601)
    at 921-ea4bddb5c25058ea.js:44:4715
    at setTable (712-08d720048b34c52c.js:1:859)
    at e (712-08d720048b34c52c.js:1:42556)
```
<img width="685" alt="Screenshot 2024-10-16 at 8 48 38 PM" src="https://github.com/user-attachments/assets/7e0549a5-f953-48c9-b0dc-867755c76c27">


oh maybe related to this issue https://github.com/promptfoo/promptfoo/issues/1731#issue-2540416297

Closing this as a duplicate of #1731.

This issue was caused by the old `app.promptfoo.dev` sharing infrastructure which stored eval results in browser localStorage. Large evals would exceed the browser's ~5MB localStorage quota, causing the `QuotaExceededError` you experienced.

We've since completely rebuilt the sharing infrastructure with the new `promptfoo.app` backend. The data is now stored server-side, so there are no more localStorage quota limitations.

If you run into any issues with the new sharing system, please open a fresh issue and we'll take a look. Thanks for reporting this!
Repository: promptfoo/promptfoo
Issue: `transformVars` not persisted and `storeOutputAs` not stored
Root cause: Documentation Deficiency

**Describe the bug**
I need to persist information between tests for threaded conversation (modifying both input and output to and from a provider). I figured the only way to do it is to use both `transformVars` and `storeOutputAs`. However, none of them work. Below I include a short reproducer.

**To Reproduce**
Just eval this (with `--verbose` to see logs):
```
description: "Send 'Hi' twice and store LLM output in myVar variable. Print myVar on each pass. On second pass it should be non-empty. But it's empty."
providers: vertex:gemini-1.5-flash
defaultTest:
  options:
    transformVars: |
      console.log("MyVar: ", vars.myVar);
      console.log("LastOutput: ", vars.lastOutput);
      
      return {
        myVar: vars.lastOutput
      };
    storeOutputAs: lastOutput
prompts: '{{ message }}{{ _conversation | reject() }}' # when _conversation is not mentioned in prompt, the tests are independent and don't share conversation (even if --max-concurrency 1)
tests:
  - vars:
      message: Hi
  - vars:
      message: Hi
  - vars:
      message: Hi
```
I use `_conversation` parameter in prompt to enforce threaded conversation to persist state. It doesn't help. Btw, one must include this parameter in the prompt! If it is mentioned only in e.g., promptRubric and not prompt, the conversation state is not persisted and individual test cases are evaluated independently.

**Expected behavior**
There are two tests in the yaml. 
1. The first test should set `lastOutput` variable via `storeOutputAs`
2. The second test should set `myVar` variable to `lastOutput` via `transformVars`
3. The third test is just to show outputs of both `lastOutput` and `myVar`, where both should be set.

However, none of the variables is set. Both are empty in every test case

**Screenshots**
This is the output with `--verbose --no-cache`:
```
..
MyVar:  undefined
LastOutput:  undefined
MyVar:  undefined
LastOutput:  undefined
MyVar:  undefined
LastOutput:  undefined
Setting concurrency to 1 because the _conversation variable is used.
..
```

**System information:**

- Promptfoo version: 0.93.3


Hi @strentom, thank you for the detailed issue and for bringing this up! `transformVars` doesn’t persist between tests because it’s primarily designed for cases like referencing and loading large files (e.g., from S3). This behavior is intentional, but I realize it could be better documented, so I’ll work on adding more clarity in the docs.

I’ll take a closer look at the `storeOutputAs` behavior today. If you want to throw some time on my calendar: https://cal.com/michael-dangelo/15min we can talk about your use case and see if there's a better way to unblock you. Thanks for your patience and for being a part of the community!

Thank you very much, I didn't know that `transformVars` is ephemeral by design. I managed to implement the necessary behavior by storing a conversation state in a custom provider. That way, I avoided `transformVars` and don't need to `storeOutputAs` in `defaultTest`.