zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Feature Request: [Web UI] Show assertion details when using assert-set
Root cause: Documentation Deficiency

Thank you for adding #765 (assert-set) and #903 (__threshold for csv). 👍 

But I found an issue while using it.
I thought below should be shown same behavior.

a.yaml
```
- vars:
      abc: "test"
- assert:
      type: assert-set
      threshold: 0.5
      assert:
        - type: python
          value: "file://a.py"
        - type: python
          value: "file://b.py
```

a.csv
| abc | __expected1 | __expected2 | __threshold |
| - | - | - | - |
| test | file://a.py | file://b.py | 0.5 |

But I got results on Web UI
1. for `a.yaml` : Just shown `Aggregate score 1 > 0.5 threshold` in overview and detail view
![image](https://github.com/promptfoo/promptfoo/assets/17977017/3b9b176b-97d8-4ea4-8267-7efdc8cd8661)

2. for `a.csv` : `Aggregate score 1 > 0.5 threshold` in overview is same with `a.yaml`, but detail view shows pass/fail details for all of assertions

Is this the intended result?
I think 1 should also show assertions details in detail view same as 2. If no, could you explain this differences?

Hey @sjsinju! Thanks for the detailed bug report – and apologies for the long delay in getting back to you!

Great news: this has been fixed! 🎉

## What was happening

You correctly identified that YAML `assert-set` and CSV `__expected` columns were behaving differently. The root cause was how assertion results were structured:

| Format | Result Structure |
|--------|------------------|
| CSV `__expected` columns | Assertions flat → results directly in `componentResults` ✓ |
| YAML `assert-set` | Assertions nested → results buried in `componentResults[0].componentResults` ✗ |

The UI reads `componentResults` to display the Evaluation panel, so with `assert-set`, it only saw the aggregate result ("Aggregate score 1 > 0.5 threshold") and missed the sub-assertion details.

## The fix

In PR #3142, we refactored the assertion result handling to automatically **flatten nested results**. Now when you use `assert-set`:

```yaml
- assert:
    type: assert-set
    threshold: 0.5
    assert:
      - type: python
        value: "file://a.py"
      - type: python
        value: "file://b.py"
```

The detail view will show:
1. The aggregate `assert-set` result with threshold info
2. **Each individual sub-assertion** with its pass/fail status, score, and reason

This matches the behavior you expected from CSV! Closing this as resolved. Thanks again for the thorough report with screenshots – it really helped track this down!
Repository: promptfoo/promptfoo
Issue: Alternative to Nunjucks?
Root cause: Configuration Error

One issue I've encountered frequently with Nunjucks is that because it is designed to work with HTML templating, it by default really wants to add HTML-escaped characters to anything it templates.

For example:

```yaml
- role: system
  content: |
    You are a helpful assistant which suggests users healthy snacks to purchase

- role: user
  content: '{{ request }}'
```

For the 'request' variable, nunjucks will HTML escape any characters that are deemed sensitive. Just a couple of examples:
- '<' becomes '&lt'
- '>' becomes '&gt'
- '"' becomes '&quot'

Given the general sensitivity of LLMs, I'd like to ensure prompts in general are as close to natural language as possible. I have noticed greater quality outputs the more natural language is used.

We can use the 'safe' keyword in some circumstances but not all. For example trying to use 'safe' in the above example could result in a malformed YAML file.

Trying to think of perhaps a specific templating alternative for YAML..

The pattern that I've been using is `{{ varname | dump | safe}}`, which will quote the string accordingly.

```yaml
- role: user
  content: {{ request | dump | safe }}
```

Maybe this needs to be more prominent in the examples?

@typpo Not sure if I understood what you're suggesting, but the issue is not just for YAML (I'm not using YAML anywhere and I still hit the "escaping" issue). IMO the tool shouldn't change the prompt in any way as LLMs are quite sensitive to such alterations.

The above answer applies to any prompt template (yaml, json, or plaintext).  But in #101 I have disabled nunjucks autoescaping, which should largely solve this problem.  Will be in the next release!

Perfect!
Repository: promptfoo/promptfoo
Issue: ValidationException: Malformed input request when using Cohere embedding with Bedrock provider
Root cause: Interface Contract Mismatch

I have the following configuration:

```
defaultTest:
  options:
    provider:
      text:
        id: bedrock:anthropic.claude-3-haiku-20240307-v1:0
        config:
          region: eu-central-1
      embedding:
        id: bedrock:embeddings:cohere.embed-english-v3
        config:
          region: eu-central-1

...
tests:
    - type: answer-relevance
      threshold: 0.9
```

I get the following error when running `promptfoo eval`:
```
API call error: ValidationException: Malformed input request: #: required key [texts] not found#: required key [input_type] not found#: extraneous key [inputText] is not permitted, please reformat your input and try again.
```

This error disappears when I'm using another Bedrock vector model such as amazon.titan-embed-text-v1 so I assume the error doesn't come from my configuration


Thanks for flagging.  The cohere embeddings request and response are different from Titan, unfortunately.  #924 should address this case based on the [AWS docs](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-embed.html).

This should be solved as of 0.64.0.  Please let me know if you run into any issues!

Hey @typpo, thanks for the quick follow-up. I'm still getting this error using 0.65.2 though 😔

Hey @typpo, I'm still getting this issue as of 0.77.0 

I'm on 0.75.2 and I'm getting 
```
API call error: ValidationException: Malformed input request: #: required key [input_type] not found, please reformat your input and try again.
```
As per the [docs](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-embed.html) the field is mandatory.
Repository: promptfoo/promptfoo
Issue: Retry only failed tests
Root cause: Documentation Deficiency

Hi, great project!

Is there support for ignoring the cache and retrying only tests that didn't pass? Something like [playwright](https://playwright.dev/docs/test-retries#retries). If not, would you be open to me opening a PR to add support for it?

Thanks! 

This is actually supported through the `--filter-failing` flag! Thanks to @mikkoh 
 https://promptfoo.dev/docs/usage/command-line/#promptfoo-eval

It takes a JSON output from a previous `eval` run, and only runs the tests that failed.   For example:
```sh
promptfoo eval -o output.json

# ...

promptfoo eval --filter-failing output.json
```

I am getting the error 
<img width="324" alt="image" src="https://github.com/promptfoo/promptfoo/assets/446402/03dc8165-1d7b-4b2e-a488-d3f4d0025ff8">


@typpo I did it exactly like this, but the tests are not running a second time.

```
promptfoo eval -o output.json
promptfoo eval --filter-failing output.json
```
Repository: promptfoo/promptfoo
Issue: [BUG] When using eval command there are diff between using config option and not using config option
Root cause: Validation Gap

I found that if i use eval command with config option, the prompts label is disappeared.

I try to analyze why it cause and found out that there is code problem.
In deail,
`main.ts ->` when using config option, it use `readconfigs` function which is using readConfig multiple time for reading config files and do post process for each config datas.
and when not using config option, it use `maybeReadConfig` function which is only using `readConfig` onetime.
The function `readConfigs`'s post processing has problem.
`util.ts ->` `readConfigs` has internal function `seenPrompts` process only prompt's id without process label.
It cause the problem that label gonna disappear when I use objects array that contain both id and label.
In result, result table miss the label option and display python code or raw prompt which did't render template variables.


Could you check it and give me a comment that whether it is intentional or not?

This is not intentional - seems to be a bug.  `seenPrompts` should take into account the label as well.  Thank you for catching and analyzing this bug!

@typpo 
It seems that Version 0.64.0 didn't have this bug patch.
Could you tell me when this fix is applied in release package?
I hope the bug patch is reflected in a released version.

Thanks @suhwan-kang, this will be included in the next release!

released in 0.65.2
Repository: promptfoo/promptfoo
Issue: [Feature suggestion]: Exposing model costs for usage with Custom Providers
GitHub labels: Feature Request

Currently I have code like this, hardcoding the model cost into my custom provider, and returning the total cost that way.

```javascript

// ...

const inputCost = 0.00025 / 1000; //@TODO: Get from provider instead?
const outputCost = 0.00125 / 1000; //@TODO: Get from provider instead?
const totalCost = inputCost * usage.promptTokens + outputCost * usage.completionTokens || undefined;

return {
  // Required
  output: object,

  // Optional
  tokenUsage: {
    total: usage.totalTokens,
    prompt: usage.promptTokens,
    completion: usage.completionTokens
  },

  cost: totalCost
};
```

From what I can see, it's not exposed currently, but it'd be super cool if I would be able to look up the model name via the provider, "claude-3-haiku-20240307", and get the cost model back to be used in the custom provider like this. 🙌

Hey Codeshark,

Cool feature request !

I was also wondering how you accomodated the custom provider did you use the [python guide](https://www.promptfoo.dev/docs/providers/python) or the [Custom Scripts](https://www.promptfoo.dev/docs/providers/custom-script)

> Hey Codeshark,
> 
> Cool feature request !
> 
> I was also wondering how you accomodated the custom provider did you use the [python guide](https://www.promptfoo.dev/docs/providers/python) or the [Custom Scripts](https://www.promptfoo.dev/docs/providers/custom-script)

Heya @SysOverdrive 🙏 you can check out the full code I'm using here: https://github.com/promptfoo/promptfoo/blob/main/examples/custom-provider/vercelAiSdkExample.js (it should be .mjs)

@Codeshark-NET Love this feature request! 

Would recommend attempting to extract $/tokens this from the API providers, as opposed to adding manually - the cost for input and output tokens seems to be changing fairly rapidly.
Repository: promptfoo/promptfoo
Issue: Avg Latency is not cached
Root cause: Unimplemented Feature Gap

Hello.
It seems like `Avg Latency` is not cached so that I cannot compare them.
I hope`Avg Latency` is cached as well.

<img width="220" alt="image" src="https://github.com/promptfoo/promptfoo/assets/62974859/73dda94b-baa4-4c40-be03-581ef4d54cbb">

FYI, My use case is "Running eval again after adding one more provider".
Repository: promptfoo/promptfoo
Issue: OpenAI Assistant - Single conversation/thread with multiple messages
Root cause: Documentation Deficiency

I'm trying to assert the result of an OpenAI Assistant as per the docs here: https://www.promptfoo.dev/docs/guides/evaluate-openai-assistants/ but can't find a way to prepend the new test messages to a previous thread/conversation, there's a way to do this?

ie:
```
prompts:
  - '{{message}}'

providers:
  - openai:assistant:...
tests:
    vars:
      message: ['Who founded Facebook?', 'When was he born?'] # the last message here is not added to the assistant thread
```

You'd have to build multiple messages into the prompt itself, for example by having a variable for previous chat messages.  See https://www.promptfoo.dev/docs/configuration/chat/.  Let me know if you run into any issues!

@typpo I was able to use the prompt in that configuration for simulating a conversation, now I have a different problem: I want to have multiple different conversations to be part of a single test suite, there's a way to do this?
Something like this:
```
description: 'Global Test Suite'

prompts: [prompt.json]

providers:
 - openai:gpt-4o

tests:
  - description: "Test Conversation 1"
    - vars:
        question: Who founded Facebook?
    - vars:
        question: Where does he live?
    - vars:
        question: Which state is that in?
  - description: "Test Conversation 2"
    - vars:
      question: Who founded Twitter?
    - vars:
      question: "Is he into Bitcoin?"
 
```

Where ideally, each vars could have its own assertion as well.
Repository: promptfoo/promptfoo
Issue: Error writing latest results
Root cause: Validation Gap

Error:
```
Failed to write latest results to C:\Users\andys-pc\.promptfoo\output\eval-2023-08-02T21-53-30.582Z.json:
Error: ENOENT: no such file or directory, lstat 'C:\Users\andys-pc\.promptfoo\output\latest.json'
```

I haven't created this directory

Could be something wrong near the `fs.symlinkSync`

also failing with this error for me on mac

I rolled back to 0.17.9 with `npm install -g promptfoo@0.17.9` and it fixed the issue.    

Thanks for the reports!  This _should_ be fixed in 0.18.1
Repository: promptfoo/promptfoo
Issue: Providers absolute files are not working
Root cause: Algorithmic Error

Hello,

I am trying to add a custom provider with an absolute path but it is not working, if I include it with a relative path and run promptfoo in the specific path then it works without problems. I am following the example of: https://www.promptfoo.dev/docs/providers/custom-api#multiple-instances-of-the-same-provider

When I use:
```
providers:
  - id: file:///Users/myuser/project/provider.js
```

I got the following error when executing promptfoo from `/Users/myuser/project`

`Serialized Error: { code: 'ERR_MODULE_NOT_FOUND', url: 'file:///Users/myuser/project/Users/myuser/project/provider.js' }`

You can see that promptfoo is ignoring the three slashes from file (file:///) and is treating the path as relative rather than absolute.


I think it has to do with this logic that is not considering that the path can already be absolute:

https://github.com/promptfoo/promptfoo/blob/13a970d2591a13e364b08ca976b2d4ec2ee3a3e1/src/util.ts#L294

Thanks for the report @danielsantiago!  Fix is in #966 

Released in 0.65.2
Repository: promptfoo/promptfoo
Issue: punycode module is deprecated
Root cause: External Dependency Breakage

```
$ node --trace-deprecation node_modules/.bin/promptfoo --version
(node:54362) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
    at node:punycode:3:9
    at BuiltinModule.compileForInternalLoader (node:internal/bootstrap/realm:398:7)
    at BuiltinModule.compileForPublicLoader (node:internal/bootstrap/realm:337:10)
    at loadBuiltinModule (node:internal/modules/helpers:96:7)
    at Module._load (node:internal/modules/cjs/loader:1063:17)
    at wrapModuleLoad (node:internal/modules/cjs/loader:212:19)
    at Module.require (node:internal/modules/cjs/loader:1297:12)
    at require (node:internal/modules/helpers:123:16)
    at Object.<anonymous> (/Users/rui/Downloads/brew/promptfoo/node_modules/whatwg-url/lib/url-state-machine.js:2:18)
    at Module._compile (node:internal/modules/cjs/loader:1460:14)
0.65.1
```

env: node 22.3.0

This is a sub-dependency of AJV that needs to be fixed upstream. You can track the fix here: https://github.com/fastify/fast-uri/issues/85 

The issue `(node:21571) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.` still persists.

The development has a fix merged for this warning, but when I am using promptfoo (installed using brew on Mac), still getting the warning.

This issue is still present in mac with the latest version.

Confirmed, built from main and has this issue.

If you want to simply hide it:

```
NODE_NO_WARNINGS=1 npx promptfoo@latest view
```

I got the same issue using Macos with npm 10.9.2 and node 22. I installed promptfoo using npm.

Still getting this.

Here's the deps tree:

```
❯ pnpm why fast-uri

devDependencies:
promptfoo 0.103.13
├─┬ ajv 8.17.1
│ └── fast-uri 3.0.1
└─┬ ajv-formats 3.0.1
  └─┬ ajv 8.17.1 peer
    └── fast-uri 3.0.1
```

Hi @Germano0 and everyone following this issue! Closing this issue as the punycode deprecation warnings were resolved a while ago. If you're still seeing this warning, please make sure you're on the latest version of promptfoo. Thanks for your patience while we addressed this Node.js ecosystem-wide issue!
Repository: promptfoo/promptfoo
Issue: [WebUI] Do not re-render markdown for images in result cells
Root cause: Architectural Constraint

Changing the layout of the eval result table in the web UI causes the result cells to re-render markdown:

https://github.com/promptfoo/promptfoo/assets/17098959/c22fc062-fc05-4ff3-9c31-6106a146618a

Interestingly it doesn't look like that happens for the variable table cells.

Thanks for flagging.  I think maybe the ReactMarkdown parts need to be put in their own component and memoized.
Repository: promptfoo/promptfoo
Issue: Not able to get vars value inside assert javascript type to implement custom accuracy algorithem
Root cause: Documentation Deficiency

**Issue Description**
- I am using node-package promptfoo prototype for prompt testing.
- I have series of vars input in the csv file with __expected output.
- expected output pattern is like . { Level 1: "Data1", Level 2: "Data2", Level 3 : " Data3"}
- I want to make an algorithm to give accuracy to each **vars** output based on the expected level output match . For example - If Level 1 matched then 50%, If Level 2 matched then 30% if Level 3 Matched then 20%.
- I am able to get the output while using the assert type `javascript` but not able to compare it with the vars input.

** What we want**
- I want to use some script after the output returned and run some matching and providing some accuracy which we can get individual prompt accuracy as well as commutative accuracy.
- I want to access vars input value from csv file to our script so we can compare the output and expected value.

Hope you will provide better guidance.

we can get by using testCase parameter in javascript assertion method.
Repository: promptfoo/promptfoo
Issue: How to run a test_use_cases.yaml on all the prompts from an input CSV.
Root cause: Documentation Deficiency

### Current Workflow:

- description: "SELECT statement with WHERE clause"
  vars:
    question: |
      Show all trades where the status is 'Open'
  assert:
    - type: icontains-all
      value: ["SELECT", "FROM", "WHERE"]
      
- description: "SELECT statement with WHERE clause"
  vars:
    question: |
      Show all trades where the status is 'Close'
  assert:
    - type: icontains-all
      value: ["SELECT", "FROM", "WHERE"]
 
### Proposed Workflow:

- description: "SELECT statement with WHERE clause"
  vars:
    question: |
      **input: input.csv::column_name**
  assert:
    - type: icontains-all
      value: ["SELECT", "FROM", "WHERE",  'Open']
      
      
  Is there any way to input a column of prompts from a csv to a set of assertions from promptfoo ?
  
  
Yours,
      

Hi @SysOverdrive,

Great question! You can absolutely do this - promptfoo supports loading test cases from CSV and applying the same assertions to all of them using `defaultTest`.

**Here's how to set it up:**

**1. Create your CSV file with the questions:**

```csv title="input.csv"
question
"Show all trades where the status is 'Open'"
"Show all trades where the status is 'Close'"
"Show all trades where the status is 'Pending'"
"Show all trades created today"
```

**2. Use `defaultTest` to apply the same assertions to all tests:**

```yaml title="promptfooconfig.yaml"
prompts:
  - "Generate SQL: {{question}}"

providers:
  - openai:gpt-4

# Apply these assertions to ALL test cases
defaultTest:
  assert:
    - type: icontains-all
      value: ["SELECT", "FROM", "WHERE"]

# Load all questions from CSV
tests: file://input.csv
```

That's it! Each row in your CSV becomes a test case, and `defaultTest` applies your assertions to all of them automatically.

**Adding test-specific assertions:**

If you need additional assertions for specific tests, you can add them in the CSV using special columns:

```csv title="input.csv"
question,__expected1
"Show all trades where the status is 'Open'","icontains: status = 'Open'"
"Show all trades where the status is 'Close'","icontains: status = 'Close'"
```

These will be **added** to the assertions from `defaultTest`.

**For more complex setups:**

You can also add metadata for filtering specific tests:

```csv title="input.csv"
question,__metadata:category,__metadata:difficulty
"Show all trades where the status is 'Open'","basic","easy"
"Show complex multi-table JOIN query","advanced","hard"
```

Then filter when running evals:
```bash
promptfoo eval --filter-metadata category=basic
```

See the [Test Cases documentation](https://www.promptfoo.dev/docs/configuration/test-cases#csv-format) for more details on CSV format and the [Configuration Reference](https://www.promptfoo.dev/docs/configuration/reference#test-case) for all `defaultTest` options.

Hope this helps! Let me know if you have any questions.
Repository: promptfoo/promptfoo
Issue: Add support for `.yml` for the config file
Root cause: Unimplemented Feature Gap

Small QOL improvement, right now the promptfoo cli does not find `promptfooconfig.yml` files, when `.yml` is a valid file descriptor according to the YAML specification. 

Done in f948f26457099db762690f48373747ad9321cc9b.  Will be in the next release!

Released in 0.66.0
Repository: promptfoo/promptfoo
Issue: How to calculate something like F1 score?
Root cause: Documentation Deficiency

If we are running classification tasks with LLM, how can we calculate overall precision, recall and F1 score from the evals?

It is not clear if derived metrics allow us to do that. Any suggestions?


Similar question in another issue - https://github.com/promptfoo/promptfoo/issues/150#issuecomment-1967994112

Check out this example: https://github.com/promptfoo/promptfoo/tree/main/examples/f-score and let us know if you have any other questions!
Repository: promptfoo/promptfoo
Issue: Latency is overriden when response is cached
Root cause: Algorithmic Error

<img width="770" alt="Skjermbilde 2024-06-21 kl  10 09 43" src="https://github.com/promptfoo/promptfoo/assets/158487250/54441310-77f6-4814-b64f-97e09c2e7489">

When a test uses cached results, the latency is currently set to that of the most recent run with cached data. 
Ideally, the latency should reflect the value from the original execution when the results were first generated.

Thank you for filing this @trondrensel Latency caching is going into our backlog. This is a duplicate of https://github.com/promptfoo/promptfoo/issues/945 We can track progress there!